13 ms·
Bcachefs may be headed out of the kernel
- guerrilla 1y agoThe drama with Linux filesystems is just nuts... It never ends.
- mschuster91 1y agoThe stakes are the highest across the entire kernel. Data that's corrupt cannot (easily) be uncorrupted.
- tpolzer 1y agoBad drivers could brick (parts of) your hardware permanently. While you should have a backup of your data anyway.
- quotemstr 1y agoAt least Kent hasn't murdered his wife
- deleted 1y ago[deleted]
- msgodel 1y agoIt's crazy people spend so much time paying attention to Hollywood celebrity drama. Opens LKML archive hoping for another Linus rant.
- rendaw 1y agoI'm sure there's just as much political allstar programmer fighting at google/apple/microsoft/whatever too, just this is done in public.
- chasil 1y agoSo the assertion is that users with (critical) data loss bugs need complete solutions for recovery and damage containment with all possible speed, and without this "last mile" effort, stability will never be achieved. The objection is the tiniest bug-fix windows get everything but the kitchen sink. These are both uncomfortable positions to occupy, without doubt.
- koverstreet 1y agoNo, the assertion is that the proper response to a bug often (and if it's high impact - always) involves a lot more than just the bugfix. And the whole reason for a filesystem's existence is to store and maintain your data, so if that is what the patch if for, yes, it should be under consideration as a hotfix. There's also the broader context: it's a major problem for stabilization if we can't properly support the people using it so they can keep testing. More context: the kernel as a whole is based on fixed time tables and code review, which it needs because QA (especially automated testing) is extremely spotty. bcachefs's QA, both automated testing and community testing, is extremely good, and we've had bugfix patchsets either held up or turn into flamewars because of this mismatch entirely too many times.
- NewJazz 1y ago[flagged]
- koverstreet 1y agoOh my. Got any other prescriptions for me? :) It's a bcachefs thread, and I'm the resident bcachefs expert, so.... :) I'm not terribly invested in these threads, the actual decisionmaking happens elsewhere. But they are a good opportunity to educate people on the whole process of shipping a filesystem, talk about what we're doing, what our priorities are, all that jazz.
- NewJazz 1y agoYes, drink lots of water and get lots of sleep lol I mean if you insist OK, but I think there are a lot better ways to educate folks.
- shmerl 1y agoMay be bcachefs should have been governed by a group of people, not a single person.
- mananaysiempre 1y agoCommittees are good-to-acceptable for keeping things going, but bad for initial design or anything requiring a coherent vision and taste. There are some examples of groups that straddled the boundary between a committee and a creative collaboration and produced good designs (Algol 60; RnRS for n ≤ 5; IIRC the design of ZFS was produced by a three-person team), but they are more of an exception, and the secret of tying together such groups remotely doesn’t seem to have been cracked. Even in the keeping things going department, a committee’s inbuilt and implicit self-preservation mechanisms can lead it to keep fiddling with things far longer than would be advisable.
- shmerl 1y agoIn this case it's more about keeping things in check and not letting one person with an attitude to ignore kernel development rules derail the whole project. I'm not saying those concerns are wrong, but when it's causing a fallout like being kicked out from the kernel, the downsides clearly are more severe than any potential benefits.
- koverstreet 1y agoActually, I think remote collaboration can work with the right medium and tools. For bcachefs, that's been IRC; we have an extremely active channel where we do a lot of collaborative debugging, design discussion, helping new users, etc. I know a lot of people heavily use slack/discord these days, but personally I find the web interfaces way too busy. IRC all the way, for me. But the problem of communicating effectively enough to produce a coherent design is very real - this goes back to Fred Brooks (Mythical Man Month). I think bcachefs turned out very well with the way the process has gone to date, and now that it's gotten bigger, with more distinct subsystems, I am very eagerly looking forward to the date when I can hand off ownership of some of those subsystems. Lately we've had some sharp developers getting involved - for the past several years it's been mainly users testing it (and some of them have gotten very good at debugging at this point). So it's happening.
- charcircuit 1y agoIf Linux would add a stable kernel module API this wouldn't be a huge a problem and it would be easy for bcachefs to ship as a kernel module with his own independent release schedule.
- josephcsible 1y agoThe slight benefit for out-of-tree module authors wouldn't be worth the negative effects on the rest of the kernel to everyone else.
- charcircuit 1y ago"slight benefit"? Having a working system after upgrading your kernel is not just a slight benefit. It's table stakes. Especially for something critical like a filesystem it should never break. >negative effects on the rest of the kernel Needing to design and support an API is not purely negative for kernel developers. It also gives a change to have a proper interface for drivers to use and follow. Take a look at the Rust for Linux which keeps running into undocumented APIs that make little sense and are just whatever <insert most popular driver> does.
- josephcsible 1y ago> Having a working system after upgrading your kernel is not just a slight benefit. It's table stakes. We already have that, with the "don't break userspace" policy combined with all of the modules being in-tree. > Needing to design and support an API is not purely negative for kernel developers. Sure, it's not purely negative, but it's overall a big net negative. > Take a look at the Rust for Linux which keeps running into undocumented APIs that make little sense and are just whatever <insert most popular driver> does. That's an argument against a stable module API! Those things are getting fixed as they get found, but if we had a stable module API, we'd be stuck with them forever. I recommend reading https://docs.kernel.org/process/stable-api-nonsense.html https://docs.kernel.org/process/stable-api-nonsense.html
- 1y ago
- dralley 1y agoI donate to Kent's patreon and I'm very enthusiastic about bcachefs. However, Kent, if you read this: please just settle down and follow the rules. Quit deliberately antagonizing Linus. The constant drama is incredibly offputting. Don't jeopardize the entire future of bcachefs over the silliest and most temporary concerns. If you absolutely must argue about some rule or other, then make that argument without having your opening move be to blatantly violate them and then complain when people call you out. You were the one who wanted into the kernel despite many suggestions that it was too early. That comes with tradeoffs. You need to figure out how to live with that, at least for a year or two. Stop making your self-imposed problems everyone else's problems.
- NewJazz 1y agoSeriously how hard is it to say "I'm unhappy users won't have access to this data recovery option but will postpone its inclusion until the next merge window". Yeah, maybe it sucks for users who want the new option or what have you, but like you said it is a temporary concern.
- vbezhenar 1y agoWhy does it suck for users? Those brave enough to use new filesystem, surely can use custom kernel for the time being, while merge effort is underway and vanilla kernel might not be the most stable option.
- cwillu 1y agoI believe part of the problem is distributions including it in their installers without requiring any of the usual “type the words “I KNOW WHAT I'M DOING”” to proceed” warning gates that are otherwise typical.
- torbid 1y agoIt seems to me like the goal is to work around the user having to type that to use Bcachefs while implying that they will have with the standard gatekeepers to avoid any limits on adoption via quality checks.
- baggy_trough 1y agoNo matter how good the code is, Overstreet's behavior and the apparent bus factor of 1 leave me reluctant to investigate this technology.
- dsp_person 1y agoCurious about this process. Can anyone submit patches to bcachefs and Kent is just the only one doing it? Is there a community with multiple contributors hacking on the features, or just Kent? If not, what could he do to grow this? And how does a single person receiving patreon donations affect the ability of a project like this to get passed bus factor of 1?
- nolist_policy 1y agoGenerally you need a maintainer for your subsystem who sends pull requests to Linus.
- deleted 1y ago[deleted]
- koverstreet 1y agoI take patches from quite a few people. If the patch looks good, I'll generally apply it. And I encourage anyone who wants to contribute to join the IRC channel. It's not a one man show, I work with a lot of people there.
- devwastaken 1y agoGood. There is no place for unstable developers in a stable kernel.
- msgodel 1y agoThe older I get the more I feel like anything other than the ExtantFS family is just silly. The filesystem should do files, if you want something more complex do it in userspace. We even have FUSE if you want to use the Filesystem API with your crazy network database thing.
- _zagj 1y ago> The older I get the more I feel like anything other than the ExtantFS family is just silly. The extended (not extant) family (including ext4) don't support copy-on-write. Using them as your primary FS after 2020 (or even 2010) is like using a non-journaling file system after 2010 (or even 2001)--it's a non-negotiable feature at this point. Btrfs has been stable for a decade, and if you don't like or trust it, there's always ZFS, which has been stable 20 years now. Apple now has AppFS, with CoW, on all their devices, while MSFT still treats ReFS as unstable, and Windows servers still rely heavily on NTFS.
- msgodel 1y agoAgain I don't really want the kernel managing a database for me like that, the few applications that need that can do it themselves just fine. (IME mostly just RDBMSs and Qemu.)
- robotnikman 1y ago>Windows will at some point have ReFS They seem to be slowly introducing it to the masses, Dev drives you set up on Windows automatically use ReFS
- milkey_mouse 1y agoHell, there's XFS if you love stability but want CoW.
- josephcsible 1y agoXFS doesn't support whole-volume snapshots, which is the main reason I want CoW filesystems. And it also stands out as being basically the only filesystem that you can't arbitrarily shrink without needing to wipe and reformat.
- criticalfault 1y agoI've been following this for a while now. Kent is in the wrong. Having a lead position in development I would kick Kent of the team. One thing is to challenge things. What Kent is doing is something completely different. It is obvious he introduced a feature, not only a Bugfix. If the rules are set in a way that rc1+ gets only Bugfixes, then this is absolutely clear what happens with the feature. Tolerating this once or twice is ok, but Kent is doing this all the time, testing Linus. Linus is absolutely in the right to kick this out and it's Kent's fault if he does so.
- Pet_Ant 1y agoWhy take it out of the kernel? Why not just make someone responsible the maintainer so they can say "no, next release" to his shenanigans? It can't be the license.
- nolist_policy 1y agoKent can appoint a suitable maintainer if he wishes. That's his job, not Linus'.
- criticalfault 1y agoThis is for me unclear as well, but I'm saying I wouldn't hold it against Linus if he did this. And based on Kent's behavior he has full right to do so. A way to handle this would be with one person (or more) in between Kent and Linus. And maybe a separate tree only for changes and fixes from bcachefs that those people in between would forward to Linus. A staging of sorts.
- tliltocatl 1y agoMaintainers aren't getting paid and so cannot be "appointed". Someone must volunteer - and most people qualified and motivated enough are already doing something else.
- timewizard 1y agoPresumably there would be an open call where people would nominate themselves for consideration. These are problems that have come up and been solved in human organizations for hundreds of years before the kernel even existed.
- layer8 1y agoFor some reason I always read this as “BCA chefs”.
- kzrdude 1y agotoday Kent posted another rc patch with a new filesystem option. But it was merged..
- ajb 1y agoYeah.. the thing is, suppose Kent was 100% right that this needed to be merged in a bugfix phase, even though it's not a bug fix. It's still a massive trust issue that he didn't flag up that the contents of his PR was well outside the expected. That means Linus has to check each of his PRs assuming that it might be pushing the boundaries without warning. No amount of post hoc justification gets you that trust back, not when this has happened multiple times now.
- NewJazz 1y agoHe mentioned it in his PR summary as a new option. About half of the summary of the original PR was talking about the new option and why it was important. https://lore.kernel.org/linux-fsdevel/4xkggoquxqprvphz2hwnir7nnuygeybf2xzpr5a4qtj4cko6fk@dlrov4usdlzm/ https://lore.kernel.org/linux-fsdevel/4xkggoquxqprvphz2hwnir...
- ajb 1y agoI'm not saying he made a PR just saying "Fixes" like a rookie. What I'm saying is that in there should have been something along the lines of "heads up - I know this doesn't comply with the usual process for the following commits, here's why I think they should be given a waiver under these circumstances" followed by the justifications that appeared after Linus got upset. The PR description would have been fine - if it had been in the right stage of the process.
- gdgghhhhh 1y agoIn this context, this is worth a read: https://hachyderm.io/@josefbacik/114755106269205960 https://hachyderm.io/@josefbacik/114755106269205960
- wmf 1y agoA lot of open source volunteers can't really be replaced because there is no one willing to volunteer to maintain that thing. This is complicated by the fact that people mostly get credit for creating new projects and no credit for maintenance. Anyone who could take over bcachefs would probably be better off creating their own new filesystem.
- deleted 1y ago[deleted]
- ajb 1y agoEhh. I don't think Kent is an arsehole. The problem with terms like "arsehole" that is that they conflate a bunch of different issues. It doesn't really have much explanatory power. Someone who is difficult to work with can be that way for loads of different reasons: Ego, tunnel vision, stress, neuro divergence (of various kinds), commercial pressures , greed, etc etc. There is always a point where you have to say "no I can't work with this person any more", but while you are still trying to it's worth trying to figure out why someone is behaving as they do.
- skissane 1y ago> The problem with terms like "arsehole" that is that they conflate a bunch of different issues. Agree, plus I’d add: if we are going to criticise other people’s communication style/abilities or attitude, then using a vague, vulgar and hurtful slang term like “arsehole”/“asshole” (and similar slang such as “dick”, “prick”, etc) is an example of exhibiting the very thing one is complaining about in making the complaint, which is fundamentally hypocritical. One can state the same concerns in a more professional way, focusing on the details of the specific behaviour pattern not a vague term which can refer to lots of distinct behaviours (e.g. people with ASD traits who hurt the feelings of others because they honestly have trouble thinking about them, versus people with antisocial or narcissistic personality disorder traits who knowingly hurt the feelings of others because they enjoy doing so) - labelling the behaviour pattern not the person, acknowledging that it is entirely possibly due to an unintentional skills gap, (sub)culture clash, differences in life experiences, neurodiversity/neurodivergence/mental health/trauma, etc. I also think it is helpful when criticising the flaws of others to try to relate them to one’s own, whenever possible - e.g. sometimes in the past I did X and from my perspective it looks like you are doing something similar-hurtful labels are not encouraging that kind of self-reflectiveness at all, they promote the idea that “I’m one of the good ones but you are one of the bad ones”
- ars 1y agoThis happened about a year ago as well: https://news.ycombinator.com/item?id=41407768 https://news.ycombinator.com/item?id=41407768
- jagged-chisel 1y agoFor the uninitiated: bCacheFS, not BCA Chefs. I’m not clued into the kernel at this level so I racked my brain a bit.
- zahlman 1y agoI had to think about it the first time, too.
- skissane 1y agoGiven the context is an OS kernel, even if I hadn't heard of "bcachefs" before (which I have), parsing it as "bcache-fs" seems obvious. By contrast, I don't know what "BCA Chefs" is supposed to be. "BCA" could be many things: "Barbados Cricket Association", "Billiard Congress of America", "British Caving Association", "Business Council of Australia", among others. But what would any of them have to do with "Chefs"?
- anonfordays 1y agoLinux needs a true answer to ZFS that's not btrfs. Sadly the ship has sailed for btrfs, after 15+ years it's still not something trustable. Apparently bcachefs won't be the successor. Filesystem development for Linux needs a big shakeup.
- bombcar 1y agoZFS is good enough for 90% of people who need that so no real money is available for anything new. Maybe a university could do it.
- anonfordays 1y agoIndeed, and it's inclusion in Ubuntu is fantastic. It's also showing it's age, 20 years now. Tso, where are you when we need you most!?
- em-bee 1y agoseveral people i know are using btrfs without problems for years now. i use it on half a dozen devices. what's your evidence that it is not trustable?
- anonfordays 1y ago
- zahlman 1y agoDoes the filesystem actually need to be part of the kernel project to work? I can see where you'd need that for the root filesystem, but even then, couldn't one migrate an existing installation to a new partition with a different filesystem?
- teekert 1y agoWe ZFS for that. What we want is something in kernel, ready to go, 100% supported on root ok any Linux system with no license ambiguity. We want to replace ext4. Maybe btrfs can do it. I hear it has outgrown its rocky puberty.
- alphazard 1y agoBcachefs and Btrfs are not really competing with Ext4. There are basically 2 filesystem niches. First niche is the full featured CoW filesystem; it has snapshots, detects and repairs corruption, transparent compression, all that good stuff. The other niche is being an allocator of sectors. There's one storage device, divide it up amongst all these processes asking for storage. That's Ext4: an allocator of disk sectors, dressed up in a filesystem API. When you are running databases or VMs, all you want is an allocator of sectors. You don't want lots of stuff getting in the way of your writes. You don't want checksumming, you don't want your writes going to a new place every time. You just want write access to part of the disk.
- deleted 1y ago[deleted]
- teekert 1y agoI want COW everywhere because I want to revert to snapshots on my laptop as much as on my servers. I want it integrated into the bootloader and boot into snapshots, I want ransomware protection on my laptop, etc.
- bombcar 1y agoI want the default file system everywhere be CoW and snapshot-enabled (a perfection would have snapshot on the file and directory level) so that tooling starts to assume it is available and begins to use it.
- alphazard 1y agoThis whole debacle is the perfect advertisement for microkernels. The only reason Kent needs to coordinate with Linus is because filesystems need to live in the kernel. FUSE is second class. Imagine how much easier this all would be if linux maintained a slowly evolving filesystem API, and all bcachefs had to do was keep up with it.
- skissane 1y agoI don’t think FUSE is deliberately a “second class citizen”, it is simply that doing a filesystem in user space has a performance cost compared to doing it in the kernel-and that is a very tricky problem to solve. Even microkernels have this problem, it is just you don’t notice it as readily because a pure microkernel doesn’t offer in-kernel filesystems as a comparator - but if you take a microkernel and transform it into a hybrid kernel by moving filesystems (and block device drivers) into kernel space, like NeXT/Apple did in turning Mach into XNU, almost certainly you are going to see tangible performance gains. Maybe this is less true with more modern microkernel designs such as L4, but even there I suspect it is still true, even if not to quite the same extent. I think the performance cost of FUSE compared to in-kernel filesystems is improving with time - FUSE with io_uring is a big step forward, but the immaturity of io_uring is an obstacle to its adoption, at least in the short-to-medium-term. I’m sure in the future we’ll see even further improvements in this area. But will we ever reach the Nirvana where FUSE equals the performance of in-kernel filesystems, or (maybe more realistically) the performance overhead has become so marginal nobody is bothered by it in practice? I’d like to think we eventually will, but it is far from certain.
- koverstreet 1y agoThere's no inherent reason why FUSE has to be noticably slower for buffered IO, it just hasn't gotten nearly enough well thought it attention. But that's starting to change, there's a lot more interest these days in a faster FUSE. Direct IO would be slower via FUSE, but L4 style IPC could solve that. It would be an interesting proposition, although not my first choice for the direction I want to go in :)
- 1y ago
- ddtaylor 1y agoI fear posting this because it's YouTube content and I don't know of the creator very well beyond these videos, but I have been following this saga a bit from this creator: https://www.youtube.com/@SavvyNik/videos https://www.youtube.com/@SavvyNik/videos He gets a few words wrong because my understanding is he covers the topic in a more broad way, but most of his coverage seems objective and factual. He does have some opinions, but I think it's closer to journalism of the LKML than an opinion piece.
- ncrmro 1y agoLots of people mentioning ZFS, which can’t do hibernation correctly as sometimes ZFS will still do some writes after that ram has suspended. Which I feel like would complete the story of here’s my mobile device that is snapshoted and backed up regularly. I wonder where bcachfs in regards to mobile snapshots and hibernation.
- koverstreet 1y agoWorks fine - my main development laptop has been bcachefs for ~8 years, I suspend it all the time :) I think there have been one or two bug reports in the past from rebalance not freezing in a timely manner (laptops don't usually use rebalance, that's usually a multi device thing), but I think they've been fixed. Send me a bug report if it's not :)
- ncrmro 1y agoSuspend works find on ZFS from what I’ve read but not hibernate where ram is written to disk for full power off
- ryao 1y agoLinux’s power management APIs that handle these things are behind GPL symbol exports. In theory, it should be fairly simple to resolve this if it were not for that. Right now, if you want hibernation with ZFS, you should use FreeBSD.
- eviks 1y agoEven if you're an absolute stickler for arbitrary guidelines, Linus can easily just enforce the rule and not merge, that's it! He already sees this FS as very experimental, so any subtle bugs remaining due to the dev not fixing them according to the process is acceptable. Inflating the drama and threatening compete removal is a hissy fit.
- kzrdude 1y agoLinus didn't threaten removal. Removing it from the kernel is apparently a topic that came up in a non-public maintainer conversation where both Kent and Linus were participating.
- teekert 1y agoIt's such a shame. We don't quite trust btrfs (but it's probably fine!), we don't quite trust the ZFS license and the fact that it is not in the Kernel (but it's mostly fine!), so Bcachefs would be so nice to have. A modern FS that one uses on their Linux root (or anywhere), as confidently as one does ext4. But what prevents it ultimately? This ... situation. It makes me sad. I didn't follow the details but I know that Linus is a reasonable person, and Kent is very thorough and delivering quality. But even if Linus was too much on the conservative-side here (but who's to judge??), please Kent, just fall in line. The alternative is nothing. Go have a beer with Linus.
- commandersaki 1y agoI just want an intree filesystem that does metadata and data checksumming, compression, encryption, and volumes/snapshots. Bcachefs was the one that ticked all of these. Such a shame to see it turn out this way.
- rob_c 1y agoGood
- Voultapher 1y agoI'm sure it can be annoying to collaborate with the other Linux people, especially if social interactions don't come easily to you - as I assume is the case for Kent - but if you want real adoption of your fs and improvement to the status quo, playing by their rules and being humble seems like the most reliably way to get there. I say that as someone that has donated to the project. I want to have a real alternative to zfs, let's please get there.
- CogitoCogito 1y agoReading through Kent Overstreet's comments, it seems totally correct to kick bcachefs out of the kernel. His comments demonstrate very clearly that he's not able to work under the constraints of the current kernel development process.