14 ms·

Bcachefs, an Introduction/Exploration

156 points by marbu 2y ago

LeoPanthera 2y ago
The "why not btrfs" line boils down to "it took a long time to be stable". That's a weird argument. Even if it's true, it is now stable, and has been for a long time. btrfs has long been my default, and I'd be wary of switching to something newer just because someone was mad that development took a long time.
- vbezhenar 2y ago
  Btrfs lost its credibility and many people would never trust it.
  - greenavocado 2y ago
    As little as one year ago I experienced damage on a lightly used btrfs root partition on my laptop. Never again. I use ext4 root and ZFS for /home for snapshots and transparent compression now, all on top of LVM
  - BSDobelix 2y ago
    So a year ago i tried to repeat my old trick damaging btrfs (as a user NOT root). Fill the volume with dd if=/dev/urandom of=./file bs=2M && sync && rm ./file then reboot the machine and yes it still works, it's not booting anymore, bravo. BTW: Even SLES SuseLinux Enterprise says use XFS for data btrfs just for the OS i wonder why
    - PlutoIsAPlanet 2y ago
      > BTW: Even SLES SuseLinux Enterprise says use XFS for data btrfs just for the OS i wonder why Because XFS is far quicker for server-related software such as databases and virtual machines, which are weak points on btrfs due to its COW model.
      - deleted 2y ago
        [deleted]
      - BSDobelix 2y ago
        Yeah and maybe additionally you want to keep your data and have a stable filesystem for them ;)
      - josephcsible 2y ago
        Doesn't "chattr +C" give you back that performance, while still letting you keep the rest of the benefits of Btrfs?
        curt15 2y ago
        Nodatacow is an ugly hack because it disables btrfs's core features for the affected data. It also should not be used with raid1.
    - yjftsjthsd-h 2y ago
      > So a year ago i tried to repeat my old trick damaging btrfs (as a user NOT root). Fill the volume with dd if=/dev/urandom of=./file bs=2M && sync && rm ./file then reboot the machine and yes it still works, it's not booting anymore, bravo. Do you know how ZFS handles that?
      - BSDobelix 2y ago
        Without any problems. No other Filesystem i tested has that problem (ext4, XFS, ZFS, NTFS, JFS, Nilfs2)
        yjftsjthsd-h 2y ago
        Good to hear:) My understanding is that it's easier to break a CoW filesystem like that because if you run out of space you can't even delete things (because that requires writing that change), so I'm not surprised that the rest (the non-CoW filesystems) did fine, but I'm happy to hear that ZFS also handles it.
- eptcyka 2y ago
  In 2019, btrfs ate all my data after a power cut. Btrfs peeps said it sounded like my SSD was at fault. Well, ZFS is still chugging along on that drive. I am not surprised btrfs took ages to stabilize, and it will take ages again before I rely on it. I’ve had previous btrfs incidents too. I think the argument against btrfs is that it was not good enough when btrfs devs told people to use it in production for ages.
  - greenavocado 2y ago
    Btrfs never actually stabilized it's still garbage compared to ZFS
    - leansensei 2y ago
      Care to substantiate that statement? It seems rather arbitrary to just say that it's garbage when it is running and has been running successfully for the vast majority of its users. It also offers two features that ZFS does not: the ability to grow a pool, and offline duplication.
      - swinglock 2y ago
        Does it even have RAID5?
        ZhongXina 2y ago
        Why should it matter? It's an extremely niche technology that's only interesting to some home users. I see no reasons why other users should care about a RAID level they're not interested in. (I don't use btrfs or any other COW filesystem because of significantly worse performance with some kinds of workloads, but it has nothing to do with maturity of any of them.)
        throw0101c 2y ago
        > Why should it matter? It's an extremely niche technology that's only interesting to some home users. I use RAID-Z2 in lots of places for bulk storages purposes (HPC). There's a reason why Ceph added erasure coding: * https://ceph.io/en/news/blog/2017/new-luminous-erasure-coding-rbd-cephfs/ https://ceph.io/en/news/blog/2017/new-luminous-erasure-codin... * https://docs.ceph.com/en/latest/rados/operations/erasure-code/ https://docs.ceph.com/en/latest/rados/operations/erasure-cod... When you're talking about PB of data, storage efficiencies add up.
- bheadmaster 2y ago
  Development taking long usually means that the model itself is too complicated to be done right in a reasonable time, which indicates that the "stable" implementation could still be buggy, but only if you stray away from the common path. It's hard to feel comfortable using such a software in a fundamental role such a file system.
  - 7e 2y ago
    In this case I think it’s the case that bcachefs has only a very small set of developers working in it.
    - jeltz 2y ago
      But that was not the case for btrfs.
  - humanfromearth9 2y ago
    I think it's not quite so simple. The problem of organising storage is at least complex, on a scale of "simple complicated complex chaotic". The inherent complexity might be impossible to reduce to something simple or even just complicated, except _maybe_ with layering (à la LVM2), each layer tackling one issue independently of the others. But then it's probably at the cost of performance and other efficiency. Each layer should work such that it does not interfere too much with the performance of other layers. Not easy. Given the rather cheap price of durable storage these days, I would favour rock solid, high quality code for storing my data, at the expense of some optimisations. Then again, I still like RAID, instantaneous snapshots, COW, encryption, xattr, resizable partitions, CRC... It's it possible to have all this with acceptable performance and simple code bricks combined and layered on top of each other?
  - dspillett 2y ago
    > which indicates that the "stable" implementation could still be buggy, but only if you stray away from the common path Or that the complexity is such that if a new bug is found, it may take a long time to be fixed because of the complexity, or it is fixed fast and has unexpected knock-on effects even for circumstances on the common path. Something that takes a long time to be declared stable/reliable because of its complexity, needs to spend a long time after that declaration without significant issues before I'll actually trust it. Things like btrfs definitely live in this category. bcachefs even won't be something I use for important storage until it has been battle-tested a bit more for a bit longer, though at this point it is much more likely to take over from my current simple ext4-on-RAID arrangement (and when/if it does, my backups might stay on ext4-on-RAID even longer).
- rubenbe 2y ago
  For me the killer feature of btrfs is "RAID 1 with different sized disks". For a small and cheap setup, this is perfect since a broken disk can be replaced with a bigger one and immediately (part of) the extra new disk space can be used. Other filesystems seem to only increase the size once all disks have been replaced with a bigger capacity one (last time I checked this was still the case for ZFS)
  - frankjr 2y ago
    Exactly. Provisioning a completely different set of disks when running out of capacity might be fine for a company but not for home office.
  - leansensei 2y ago
    That, plus offline deduplication.
  - e12e 2y ago
    How does that work? You have two 100gb drives in raid1, 80% full, you replace one with a 200gb disk and write 50gb to the array - how is your 130gb of data protected against either drive failing?
    - rubenbe 2y ago
      It only works with 3+ disks. All data needs to be on two disks. e.g. you have 3 100GB drives, total capacity in raid 1 is 150GB. If you replace a broken one with a 200GB one, the total capacity will be increased to 200GB.
      - assbuttbuttass 2y ago
        My understanding is that RAID1 is just a mirror, and all disks have identical contents. Are you talking about something else?
        tuetuopay 2y ago
        Traditional RAID1 will mirror whole drives, yes. BTRFS RAID1 will mirror chunks of data (iirc 1GB) on two drives. So you can have e.g. two 1TB drive and a 2TB one just fine.
- GrayShade 2y ago
  One interesting titbit I've only recently found out is that btrfs can't really serve reads from different drives in RAID1, it picks a drive based on the process id. ZFS does something smarter here, it keeps track of the queue length for each drive in a mirror, and picks the one with the lowest number of pending requests.
- _flux 2y ago
  My personal grievances with btrfs are multifaceted. - I never agreed with the btrfs default of root raid 1 system not booting up if a device is missing. I think the point of raid1 is to minimize downtime when losing a device and if you lose the other device before returning it to good state, that's 100% on you. - Poor management tools compared to md (though bcachefs might be in the same boat). Some tools are poorly thought, e.g. there is a tool for defragmentation, but it undoes sharing (so snapshots and dedupped files get expanded). - If a drive in raid1 drops but then later comes back, btrfs is still quite happy. - Need of using btrfs balance, and in a certain way as well: https://github.com/kdave/btrfsmaintenance/blob/master/btrfs-balance.sh https://github.com/kdave/btrfsmaintenance/blob/master/btrfs-... . - At least it used to be difficult to recover when your filesystem becomes full. Helps if you have it on LVM volume with extra space. - Snapshotting or having a clone of a btrfs volume is dangerous (due to the uuid-based volume participant scanning) - I believe raid5/6 is still experimental? - I've lost a filesystem to btrfs raid10 (but my backups are good). - I have also rendered my bcachefs in a state where I could no longer write to the filesystem, but I was still able to read it. So I'm inclined to keep using bcachefs for the time being. Overall I just have the impression that btrfs was complicated and ended up in a design dead-end, making improvements from hard to difficult, and I hope that bcachefs has made different base designs, making future improvements easier. Yes, the number of developers for bcachefs is smaller, but frankly as long as it's possible for a project to advance with a single developer, it is going to be the most effective way to go—at the same time I hope this situation improves in the future.
  - viraptor 2y ago
    > I never agreed with the btrfs default of root raid 1 system not booting up if a device is missing. Add "degraded" to default mount options. Solved.
    - cherryteastain 2y ago
      Good luck doing that after the disk shuts down okay but never comes back online
      - 2y ago
- qalmakka 2y ago
  Btrfs is still unacceptably less reliable than ZFS, after _decades_ of development. This is unacceptable, IMHO. I've lost so much data due to btrfs corruption issues that I've (almost) stopped to use it completely nowadays. It's better to fight to keep the damned OpenZFS modules up to date and get an actual _reliable_ system instead of accepting the risk again.
  - BSDobelix 2y ago
    >It's better to fight to keep the damned OpenZFS modules up to date and get an actual _reliable_ system Try CachyOS (or at least the ZFS-Kernel) it has excellent ZFS integration.
    - Volundr 2y ago
      This. I may still give up on running ZFS on Linux due to the common (seemingly intentional from the Linux side) breakage, but for my existing systems switching them over to CachyOS repos has been a blessed relief.
      - BSDobelix 2y ago
        Well i use mainly FreeBSD but have used CachyOS for about 3mo to have some systemd refresher :)
    - magicalhippo 2y ago
      Hadn't heard of CachyOS before, looks very nice! Was looking to move to Arch from KDE Neon, but this might be a much better fit.
      - BSDobelix 2y ago
        Well or don't move from arch and just use the cachyos repos: https://wiki.cachyos.org/de/cachyos_repositories/how_to_add_cachyos_repo/ https://wiki.cachyos.org/de/cachyos_repositories/how_to_add_... No reinstall needed ;)
- pantalaimon 2y ago
  btrfs has still many weird issues. e.g. you can't remove a drive if it has I/O errors, even if the rest of the array has still enough space to accompany the data. You can do a replace, but then you need to buy a new drive.
- nialv7 2y ago
  btrfs is not stable, at least not for me. it lost my data only a couple months ago. no power cut, no disk failure, data just gone.
- linsomniac 2y ago
  It's not simply that it took a long time to become stable; it's that during this time where it was unstable a lot of people got exposed to btrfs by having it lose data. Personally, I was one of those people. Very excited about the prospects of btrfs, switched several machines over to it to test, ended up with filesystem corruption and had to revert to ext. Now, whenever I peek at btrfs, I never see anything that's compelling over running ZFS, which I've run for close to 15+ years, and run hard, and have never had data loss. Even in the early days with zfs+fuse, where I could regularly crash the zfs fuse; the zfs+fuse developers quickly addressed every crash I ran into, once I put together a stress test.
- 112233 2y ago
  > it is now stable, and has been for a long time. Is it really? I must have missed the news. Back when it was released completely raw as a default for many distros, there were fundamental design level issues (e.g. "unbound internal fragmentation" reported by Shishkin). Plus all the reports and personal experiences of getting and trying to recover exotically shaped bricks when volume fills to 100% (which could happen at any time with btrfs). Is it all good now? Where can I read about btrfs behaving robustly when no free space is left?
ysleepy 2y ago
I wonder why ZFS is marked as not having de-dupe (deduplication). AFAIK ZFS has had deduplication support for a very long time (2009) and now even does opportunistic block cloning with much less overhead.
- adrian_b 2y ago
  Also XFS has deduplication now, already for some time, at least one year or two.
  - BlackLotus89 2y ago
    btrfs has deduplication as well. In theory full file deduplication exists in every filesystem that has cow/reflink support
    - Tobu 2y ago
      fclones for example covers it well for any filesystem with reflinks: https://lib.rs/crates/fclones https://lib.rs/crates/fclones fclones group |fclones dedupe
- ksec 2y ago
  Yes I sort of skip-read a lot of it after that.
- the8472 2y ago
  ZFS online deduplication is not comparable with on-demand dedup offered btrfs and xfs and is prohibitively expensive for many workloads. The new block cloning still had data corruption bugs quite recently.
  - BSDobelix 2y ago
    >ZFS online deduplication is not comparable with on-demand dedup offered btrfs and xfs But it has de-duplication, with your logic no non-CoW FS should be in that list because they are not comparable.
xxmarkuski 2y ago
I'm running bcache, with lvm/luks and xfs on top, since >5 years on my desktop and it has been stable and partition manipulations, like resizes, worked without problems, albeit the tooling is not so well supported. I bought new a new ssd and hdd for my desktop this year and looked into running bcachefs because it offers caching as well as native encryption and cow. I also determined that it is not production ready yet for my use case, my file system is the last thing I want to beta tester of. Investigated using bcache again, but opted to use lvm caching, as it offers better tooling and saves on one layer of block devices (with luks and btrfs on top). Performance is great and partition manipulations also worked flawless. Hopefully bcachefs gains more traction and will be ready for production use, as it combines several useful features. My current setup still feels like making compromises.
NKosmatos 2y ago
Mandatory xkcd comic: https://xkcd.com/927 https://xkcd.com/927 (replace "standards" with "FS")
- tjoff 2y ago
  Situation is different though, we have very few modern filesystems and have desperately needed some diversion and competition in this area. I've been waiting decades for something like bcachefs, thought it would be btrfs but that turned into a disappointment - for my needs.
  - leetnewb 2y ago
    It seems like "modern filesystem" development focus shifted to distributed a few years back.
- _flux 2y ago
  The key difference is that we don't need to agree on a certain FS, whereas the reason for standards is interoperability. I have both bcachefs and ext4 filesystems on the same machine, for different uses.
- jeltz 2y ago
  There are only two competitors to bcachefs: btrfs and zfs. So having a third player in this space is a good thing, especially since a lot of people (in my opinion for good reason) do not trust btrfs meaning there is only really zfs.
  - PlutoIsAPlanet 2y ago
    ZFS isn't a real competitor given it's not in the kernel and has legal troubles.
    - ZhongXina 2y ago
      Who is downvoting this? Among the large Linux distributions, ZFS is only really supported by Ubuntu, and even that is on the level "Canonical lawyers reviewed this and believe they're safe". If the unmentionable company ever goes to court against them, you're in hot water. You'll have to migrate to FreeBSD or support yourself by building dkms modules. So you're taking a non-zero risk by adopting ZFS. If you're really conservative with these things, as some of us are, you currently don't really have a single safe COW pick. (Smug FreeBSD users incoming.) I have most trust in bcachefs over the long term.
guenthert 2y ago
Hmmh, under "Why bcachefs?" we find - Stability but also - Constant refactorings and later "Disclaimer, my personal data is stored on ZFS" A bit troubling, I find "RAID0 behavior is default when using multiple disks" never have I ever had the need for RAID0 or have I seen a customer using it. I think it was at one time popular with gamers before SSDs became popular and cheap. "RAID 5/6 (experimental) This is referred to as erasure coding and is listed as “DO NOT USE YET”, " Well, you got to start somewhere, but a comparison with btrfs and ZFS seems premature.
- GrayShade 2y ago
  Does it? From the btrfs docs: > The RAID56 feature provides striping and parity over several devices, same as the traditional RAID5/6. There are some implementation and design deficiencies that make it unreliable for some corner cases and the feature should not be used in production, only for evaluation or testing. The power failure safety for metadata with RAID56 is not 100%.
- nextaccountic 2y ago
  > "Disclaimer, my personal data is stored on ZFS" > A bit troubling, I find I appreciated the candor The approach of bcachefs developers is that they will only recommend it's usage if it's absolutely, 100% stable and won't eat your data. Bcachefs isn't in that state yet and the developers don't pretend it is. This avoids the kind of trust issues that btrfs has
Liftyee 2y ago
An interesting analysis. I can't stop my brain from parsing the title as "B C A Chefs".
ajb 2y ago
Bcachefs was merged into the kernel only months ago, and had an immediate flurry of bug fixes due to the additional testing this brought. (It was in development for some years before that out of tree). That's the level of maturity that it is at. I think there's a hope that it will become more trustworthy than btrfs due to the developers success with bcache.
- snapplebobapple 2y ago
  I've been running bcachefs on my main desktop and laptop (My really important data is on my fileserver or in my private git repo (including dotfiles), I'm not crazy) since cachyos made it an install option and it's honestly worked better and caused less problems than btrfs has for me in the past so far. Maybe I was just unlucky but btrfs caused read only filesystem issues and a catastrophic loss on a couple of my computers a few years ago. I am pretty impressed with bcachefs so far.
  - bscphil 2y ago
    > since cachyos made it an install option and it's honestly worked better and caused less problems 0 problems in 2.5 months is not necessarily better than 1-2 problems in ~3 years, though. If we're just talking about the single partition boot drive use case, I think I'd go with the option that's had vastly more time to find and eliminate bugs. (If you're conservative about this stuff that probably means ext4, actually.)
    - snapplebobapple 2y ago
      I would agree with you if I didn't run my stuff the way I do (and I'd use zfs or maybe ext4). Pretty much all my important stuff is on a raidz6 file server with a secondary backup raidz6 file server locally pulling backups each night and backups being sent offsite streaming throughout the day. My dotfiles are synced to my local private git repo via yadm (although if I didn't have this system running already I would probably take the time to figure out nix home manager instead of yadm right now). I have a bunch of bash scripts I wrote to automate the most annoying parts of reinstalling as well, so what I am really risking by running bcachefs is about a half hour to reinstall cachyos on whatever system eats it and possibly some minor configuration changes I may not have synced to git via yadm yet.
frankjr 2y ago
> btrfs Encryption Y btrfs doesn't have a built-in encryption. > ZFS Encryption Y I cannot find the discussion right now but I remember reading that they were considering a warning when enabling encryption because it was not really stable and people were running into crashes. https://github.com/openzfs/zfs/issues?q=is%3Aissue+label%3A%22Component%3A+Encryption%22+is%3Aopen+label%3A%22Type%3A+Defect%22 https://github.com/openzfs/zfs/issues?q=is%3Aissue+label%3A%...
- prmoustache 2y ago
  That bug is old, is missing information and hasn't been closed while the reporter say the problem has been solved after an update. I see it more as an administrative problem than an issue with ZFS encryption.
  - GrayShade 2y ago
    See https://gist.github.com/rincebrain/622ee4991732774037ff44c6768085ab#encryption https://gist.github.com/rincebrain/622ee4991732774037ff44c67... though.
- nialv7 2y ago
  Anecdote: I've been using ZFS encryption for a looong time and never had any problems.
sevg 2y ago
I recently tried btrfs on a new USB thumb drive. I immediately got hard freezes of my main (Linux) OS while working with the USB stick. Never again. I eagerly await bcachefs reaching maturity!
- viraptor 2y ago
  I hope you reported the issue. That smells like a bug beyond the scope of btrfs itself. The basic filesystem has been stable for a very long time.
- nextaccountic 2y ago
  Maybe that was a bad USB port? (I have one such port that intermittently disconnects) I have a USB stick with btrfs + LUKS on Arch Linux and it never had a problem like this
  - sevg 2y ago
    Same port and USB stick worked fine with XFS and ext4. Tried again with btrfs and hard freezes again.
    - Rinzler89 2y ago
      >Same port and USB stick worked fine with XFS and ext4. None of those file systems are not comparable to BTRFS since they're not COW. BTRFS isn't for crappy USB drives since it has a lot more overhead than EXT4 and XFS which the controllers and flash chips in junky USB drives can't handle.
      - sevg 2y ago
        It's not a crappy USB drive. It's a high-end high-performance Sandisk USB drive. Regardless, I still expect the choice of filesystem to not hard freeze my OS. I've had janky crappy USB drives before and with any other filesystem reads/writes might fail but I don't get a hard freeze. One could argue it could be a bug in the Linux USB stack rather than a bug in the btrfs kernel driver. But what I do know is that when I pick btrfs I get problems.
amluto 2y ago
Here’s my pet peeve regarding RAID: no RAID system I’ve ever used gracefully handles disks that come and go. Concretely: start with two disks in RAID1. Remove one. Mount in degraded mode. Write to a file. Unmount. Reconnect the removed disk. Mount again with both disks. The results vary between annoying (need to restore / “resilver” and have no redundancy until it’s done; massively increased risk of data loss while doing so due to heavy IO load without redundancy and pointless loss of the redundancy that already exists) to catastrophic (outright corruption). The corollary is that RAID invariably works poorly with disks connected over using an interface that enumerates slowly or unreliably. Yet most competent active-active database systems have no problems with this scenario! I would love to see a RAID system that thinks of disks as nodes, properly elects leaders, and can efficiently fast-forward a disk that’s behind. A pile of USB-connected drives would work perfectly, would come up when a quorum was reached, and would behave correctly when only a varying subset of disks is available. Bonus points for also being able to run an array that spans multiple computers efficiently, but that would just be icing on the cake.
- TheDong 2y ago
  Does ceph not fulfill your requirements here? Especially that last "spans multiple computers" bit.
  - amluto 2y ago
    Ceph doesn’t really nail the “I want to boot off this thing” use case. It would be interesting to try, though.
    - magicalhippo 2y ago
      Ceph provides S3-compatible object store no? If so, just use s3backer[1] with a loopback mount and boot[2] off it? I mean, sounds like a house of cards but, should be possible? [1]: https://github.com/archiecobbs/s3backer https://github.com/archiecobbs/s3backer [2]: https://ersei.net/en/blog/fuse-root https://ersei.net/en/blog/fuse-root
curt15 2y ago
How well does bcachefs handle databases and VMs? Those workloads are well-known to be btrfs' kryptonite whereas ZFS seems to tolerate them pretty well as long as one sets the correct recordsize (example: https://www.enterprisedb.com/blog/postgres-vs-file-systems-performance-comparison https://www.enterprisedb.com/blog/postgres-vs-file-systems-p...).
- yarg 2y ago
  https://www.phoronix.com/review/bcachefs-benchmarks-linux67 https://www.phoronix.com/review/bcachefs-benchmarks-linux67
  - dralley 2y ago
    It's worth mentioning that bcachefs has gotten a fair bit of performance work since the last round of Phoronix benchmarks. Also there's some bug where the formatting tool selected 512 byte blocks by default instead of 4k byte blocks on drives where other filesystems picked 4k bytes, which impacted the Phoronix benchmarks. Unsure if that has been fixed yet.
- mastax 2y ago
  IIRC bcachefs is going to add a non-COW mode which should be good for databases.
  - curt15 2y ago
    NoCOW on btrfs is a kludge because it disables the core features of btrfs and is dangerous to use with raid1. Since, ZFS doesn't even have a nocow mode, surely there are other ways of dealing with databases?
linsomniac 2y ago
I had really high hopes for HAMMER2, including that it would one day be ported to Linux, but it seems to have remained firmly planted in Dragonfly BSD and it's not really clear what the status is. https://en.wikipedia.org/wiki/HAMMER2 https://en.wikipedia.org/wiki/HAMMER2
gigatexal 2y ago
> ZFS, pioneering COW filesystem, ... commendable, its block-based design diverges from modern extent-based systems due to complexities in implementing extents with snapshots. why is this a bad thing?
jcalvinowens 2y ago
The idea that a brand new filesystem might be more reliable than good 'ol BTRFS, which Facebook runs on basically their entire infrastructure, is downright laughable to me. Btrfs is also far more reliable than ZFS in my view, because it has far far more real world testing, and is also much more actively developed. Magical perfect elegant code isn't what makes a good filesystem: real world testing, iteration, and bugfixing is. BTRFS has more of that right now than anything else ever has.
- jeltz 2y ago
  Why is that laughable? I do not think that it is more reliable than btrfs but it is not a crazy idea either. There are a whole bunch of people in these comments with very recent btrfs reliability issues which have affected them and nobody with recent zfs reliability issues.
  - jcalvinowens 2y ago
    If anecdotes are meaningful (dubious, but I'll play along...), I can counter with mine: I've been running btrfs on bleeding edge kernels for a decade, and I've never seen a single data loss event. ZFS has corruption bugs, this one was far worse than anything I've seen in btrfs recently: https://lists.freebsd.org/archives/freebsd-stable/2023-November/001726.html https://lists.freebsd.org/archives/freebsd-stable/2023-Novem...
    - tiberious726 2y ago
      I've been running it on hundreds of servers in prod since once of the lead devs gave a talk at LinuxCon 2014 saying it was good to go. Had a few performance issues here and there, especially on older kernels, but never any data loss
    - bjoli 2y ago
      I also think you can't really compare them: ZFS more or less says "never use without ECC memory". BTRFS is run on just about any potato there is. I myself would never run a file server without ECC and a UPS configured for a graceful shutdown. I have also never had any issues, but I only have about 10tb of data.
Tobu 2y ago
> Error handling on CRC read error > 2 or more copies of file, CRC on error, read other copy, data returned to userspace, does not correct bad copy That's been implemented; in Linux 6.11 bcachefs will correct errors on read. See > - Self healing on read IO/checksum error in https://lore.kernel.org/linux-bcachefs/73rweeabpoypzqwyxa7hld7tnkskkaotuo3jjfxnpgn6gg47ly@admkywnz4fsp/ https://lore.kernel.org/linux-bcachefs/73rweeabpoypzqwyxa7hl... Making it possible to scrub from userspace by walking and reading everything (tar -c /mnt/bcachefs >/dev/null).
- amtadt 2y ago
  Self healing is dangerous because it can potentially corrupt good data on disk, if RAM or other system component is flaky. Repro: supposedly only good copy is copied to ram, ram corrupts bit, crc is recalculated using corrupted but, corrupted copy is written back to disk(s).
  - cesarb 2y ago
    > crc is recalculated using corrupted bit Why would it need to recalculate the CRC? The correct CRC (or other hash) for the data is already stored in the metadata trees; it's how it discovered that the data was corrupted in the first place. If it writes back corrupted data, it will be detected as corrupted again the next time.
    - amtadt 2y ago
      Because CRC is in the on-disk data structure, not in the in-ram data structure. It is stripped upon reading to ram, and created upon writing to disk. That's how bcachefs is designed right now.
      - koverstreet 2y ago
        No, we carefully carry around existing checksums when moving data. Page cache is a different story, but doesn't apply to what we're talking about here.
  - 2y ago
whalesalad 2y ago
Had to do a double-take on the UI of this blog. It looks identical to my notetaking app, Trilium.
tripdout 2y ago
Does it allow both shrinking and growing the FS? Really wish ZFS allowed shrinking.
commandersaki 2y ago
I’m pretty keen to try bcachefs. Has anyone successfully set it up as root filesystem on a raspberry pi?

156 points by marbu 2y ago

The "why not btrfs" line boils down to "it took a long time to be stable". That's a weird argument. Even if it's true, it is now stable, and has been for a long time. btrfs has long been my default, and I'd be wary of switching to something newer just because someone was mad that development took a long time.

vbezhenar 2y ago
Btrfs lost its credibility and many people would never trust it.
- greenavocado 2y ago
  As little as one year ago I experienced damage on a lightly used btrfs root partition on my laptop. Never again. I use ext4 root and ZFS for /home for snapshots and transparent compression now, all on top of LVM
- BSDobelix 2y ago
  So a year ago i tried to repeat my old trick damaging btrfs (as a user NOT root). Fill the volume with dd if=/dev/urandom of=./file bs=2M && sync && rm ./file then reboot the machine and yes it still works, it's not booting anymore, bravo. BTW: Even SLES SuseLinux Enterprise says use XFS for data btrfs just for the OS i wonder why
  - PlutoIsAPlanet 2y ago
    > BTW: Even SLES SuseLinux Enterprise says use XFS for data btrfs just for the OS i wonder why Because XFS is far quicker for server-related software such as databases and virtual machines, which are weak points on btrfs due to its COW model.
    - deleted 2y ago
      [deleted]
    - BSDobelix 2y ago
      Yeah and maybe additionally you want to keep your data and have a stable filesystem for them ;)
    - josephcsible 2y ago
      Doesn't "chattr +C" give you back that performance, while still letting you keep the rest of the benefits of Btrfs?
      - curt15 2y ago
        Nodatacow is an ugly hack because it disables btrfs's core features for the affected data. It also should not be used with raid1.
  - yjftsjthsd-h 2y ago
    > So a year ago i tried to repeat my old trick damaging btrfs (as a user NOT root). Fill the volume with dd if=/dev/urandom of=./file bs=2M && sync && rm ./file then reboot the machine and yes it still works, it's not booting anymore, bravo. Do you know how ZFS handles that?
    - BSDobelix 2y ago
      Without any problems. No other Filesystem i tested has that problem (ext4, XFS, ZFS, NTFS, JFS, Nilfs2)
      - yjftsjthsd-h 2y ago
        Good to hear:) My understanding is that it's easier to break a CoW filesystem like that because if you run out of space you can't even delete things (because that requires writing that change), so I'm not surprised that the rest (the non-CoW filesystems) did fine, but I'm happy to hear that ZFS also handles it.
eptcyka 2y ago
In 2019, btrfs ate all my data after a power cut. Btrfs peeps said it sounded like my SSD was at fault. Well, ZFS is still chugging along on that drive. I am not surprised btrfs took ages to stabilize, and it will take ages again before I rely on it. I’ve had previous btrfs incidents too. I think the argument against btrfs is that it was not good enough when btrfs devs told people to use it in production for ages.
- greenavocado 2y ago
  Btrfs never actually stabilized it's still garbage compared to ZFS
  - leansensei 2y ago
    Care to substantiate that statement? It seems rather arbitrary to just say that it's garbage when it is running and has been running successfully for the vast majority of its users. It also offers two features that ZFS does not: the ability to grow a pool, and offline duplication.
    - swinglock 2y ago
      Does it even have RAID5?
      - ZhongXina 2y ago
        Why should it matter? It's an extremely niche technology that's only interesting to some home users. I see no reasons why other users should care about a RAID level they're not interested in. (I don't use btrfs or any other COW filesystem because of significantly worse performance with some kinds of workloads, but it has nothing to do with maturity of any of them.)
        throw0101c 2y ago
        > Why should it matter? It's an extremely niche technology that's only interesting to some home users. I use RAID-Z2 in lots of places for bulk storages purposes (HPC). There's a reason why Ceph added erasure coding: * https://ceph.io/en/news/blog/2017/new-luminous-erasure-coding-rbd-cephfs/ https://ceph.io/en/news/blog/2017/new-luminous-erasure-codin... * https://docs.ceph.com/en/latest/rados/operations/erasure-code/ https://docs.ceph.com/en/latest/rados/operations/erasure-cod... When you're talking about PB of data, storage efficiencies add up.
bheadmaster 2y ago
Development taking long usually means that the model itself is too complicated to be done right in a reasonable time, which indicates that the "stable" implementation could still be buggy, but only if you stray away from the common path. It's hard to feel comfortable using such a software in a fundamental role such a file system.
- 7e 2y ago
  In this case I think it’s the case that bcachefs has only a very small set of developers working in it.
  - jeltz 2y ago
    But that was not the case for btrfs.
- humanfromearth9 2y ago
  I think it's not quite so simple. The problem of organising storage is at least complex, on a scale of "simple complicated complex chaotic". The inherent complexity might be impossible to reduce to something simple or even just complicated, except _maybe_ with layering (à la LVM2), each layer tackling one issue independently of the others. But then it's probably at the cost of performance and other efficiency. Each layer should work such that it does not interfere too much with the performance of other layers. Not easy. Given the rather cheap price of durable storage these days, I would favour rock solid, high quality code for storing my data, at the expense of some optimisations. Then again, I still like RAID, instantaneous snapshots, COW, encryption, xattr, resizable partitions, CRC... It's it possible to have all this with acceptable performance and simple code bricks combined and layered on top of each other?
- dspillett 2y ago
  > which indicates that the "stable" implementation could still be buggy, but only if you stray away from the common path Or that the complexity is such that if a new bug is found, it may take a long time to be fixed because of the complexity, or it is fixed fast and has unexpected knock-on effects even for circumstances on the common path. Something that takes a long time to be declared stable/reliable because of its complexity, needs to spend a long time after that declaration without significant issues before I'll actually trust it. Things like btrfs definitely live in this category. bcachefs even won't be something I use for important storage until it has been battle-tested a bit more for a bit longer, though at this point it is much more likely to take over from my current simple ext4-on-RAID arrangement (and when/if it does, my backups might stay on ext4-on-RAID even longer).
rubenbe 2y ago
For me the killer feature of btrfs is "RAID 1 with different sized disks". For a small and cheap setup, this is perfect since a broken disk can be replaced with a bigger one and immediately (part of) the extra new disk space can be used. Other filesystems seem to only increase the size once all disks have been replaced with a bigger capacity one (last time I checked this was still the case for ZFS)
- frankjr 2y ago
  Exactly. Provisioning a completely different set of disks when running out of capacity might be fine for a company but not for home office.
- leansensei 2y ago
  That, plus offline deduplication.
- e12e 2y ago
  How does that work? You have two 100gb drives in raid1, 80% full, you replace one with a 200gb disk and write 50gb to the array - how is your 130gb of data protected against either drive failing?
  - rubenbe 2y ago
    It only works with 3+ disks. All data needs to be on two disks. e.g. you have 3 100GB drives, total capacity in raid 1 is 150GB. If you replace a broken one with a 200GB one, the total capacity will be increased to 200GB.
    - assbuttbuttass 2y ago
      My understanding is that RAID1 is just a mirror, and all disks have identical contents. Are you talking about something else?
      - tuetuopay 2y ago
        Traditional RAID1 will mirror whole drives, yes. BTRFS RAID1 will mirror chunks of data (iirc 1GB) on two drives. So you can have e.g. two 1TB drive and a 2TB one just fine.
GrayShade 2y ago
One interesting titbit I've only recently found out is that btrfs can't really serve reads from different drives in RAID1, it picks a drive based on the process id. ZFS does something smarter here, it keeps track of the queue length for each drive in a mirror, and picks the one with the lowest number of pending requests.
_flux 2y ago
My personal grievances with btrfs are multifaceted. - I never agreed with the btrfs default of root raid 1 system not booting up if a device is missing. I think the point of raid1 is to minimize downtime when losing a device and if you lose the other device before returning it to good state, that's 100% on you. - Poor management tools compared to md (though bcachefs might be in the same boat). Some tools are poorly thought, e.g. there is a tool for defragmentation, but it undoes sharing (so snapshots and dedupped files get expanded). - If a drive in raid1 drops but then later comes back, btrfs is still quite happy. - Need of using btrfs balance, and in a certain way as well: https://github.com/kdave/btrfsmaintenance/blob/master/btrfs-balance.sh https://github.com/kdave/btrfsmaintenance/blob/master/btrfs-... . - At least it used to be difficult to recover when your filesystem becomes full. Helps if you have it on LVM volume with extra space. - Snapshotting or having a clone of a btrfs volume is dangerous (due to the uuid-based volume participant scanning) - I believe raid5/6 is still experimental? - I've lost a filesystem to btrfs raid10 (but my backups are good). - I have also rendered my bcachefs in a state where I could no longer write to the filesystem, but I was still able to read it. So I'm inclined to keep using bcachefs for the time being. Overall I just have the impression that btrfs was complicated and ended up in a design dead-end, making improvements from hard to difficult, and I hope that bcachefs has made different base designs, making future improvements easier. Yes, the number of developers for bcachefs is smaller, but frankly as long as it's possible for a project to advance with a single developer, it is going to be the most effective way to go—at the same time I hope this situation improves in the future.
- viraptor 2y ago
  > I never agreed with the btrfs default of root raid 1 system not booting up if a device is missing. Add "degraded" to default mount options. Solved.
  - cherryteastain 2y ago
    Good luck doing that after the disk shuts down okay but never comes back online
    - 2y ago
qalmakka 2y ago
Btrfs is still unacceptably less reliable than ZFS, after _decades_ of development. This is unacceptable, IMHO. I've lost so much data due to btrfs corruption issues that I've (almost) stopped to use it completely nowadays. It's better to fight to keep the damned OpenZFS modules up to date and get an actual _reliable_ system instead of accepting the risk again.
- BSDobelix 2y ago
  >It's better to fight to keep the damned OpenZFS modules up to date and get an actual _reliable_ system Try CachyOS (or at least the ZFS-Kernel) it has excellent ZFS integration.
  - Volundr 2y ago
    This. I may still give up on running ZFS on Linux due to the common (seemingly intentional from the Linux side) breakage, but for my existing systems switching them over to CachyOS repos has been a blessed relief.
    - BSDobelix 2y ago
      Well i use mainly FreeBSD but have used CachyOS for about 3mo to have some systemd refresher :)
  - magicalhippo 2y ago
    Hadn't heard of CachyOS before, looks very nice! Was looking to move to Arch from KDE Neon, but this might be a much better fit.
    - BSDobelix 2y ago
      Well or don't move from arch and just use the cachyos repos: https://wiki.cachyos.org/de/cachyos_repositories/how_to_add_cachyos_repo/ https://wiki.cachyos.org/de/cachyos_repositories/how_to_add_... No reinstall needed ;)
pantalaimon 2y ago
btrfs has still many weird issues. e.g. you can't remove a drive if it has I/O errors, even if the rest of the array has still enough space to accompany the data. You can do a replace, but then you need to buy a new drive.
nialv7 2y ago
btrfs is not stable, at least not for me. it lost my data only a couple months ago. no power cut, no disk failure, data just gone.
linsomniac 2y ago
It's not simply that it took a long time to become stable; it's that during this time where it was unstable a lot of people got exposed to btrfs by having it lose data. Personally, I was one of those people. Very excited about the prospects of btrfs, switched several machines over to it to test, ended up with filesystem corruption and had to revert to ext. Now, whenever I peek at btrfs, I never see anything that's compelling over running ZFS, which I've run for close to 15+ years, and run hard, and have never had data loss. Even in the early days with zfs+fuse, where I could regularly crash the zfs fuse; the zfs+fuse developers quickly addressed every crash I ran into, once I put together a stress test.
112233 2y ago
> it is now stable, and has been for a long time. Is it really? I must have missed the news. Back when it was released completely raw as a default for many distros, there were fundamental design level issues (e.g. "unbound internal fragmentation" reported by Shishkin). Plus all the reports and personal experiences of getting and trying to recover exotically shaped bricks when volume fills to 100% (which could happen at any time with btrfs). Is it all good now? Where can I read about btrfs behaving robustly when no free space is left?

ysleepy 2y ago

I wonder why ZFS is marked as not having de-dupe (deduplication). AFAIK ZFS has had deduplication support for a very long time (2009) and now even does opportunistic block cloning with much less overhead.

adrian_b 2y ago
Also XFS has deduplication now, already for some time, at least one year or two.
- BlackLotus89 2y ago
  btrfs has deduplication as well. In theory full file deduplication exists in every filesystem that has cow/reflink support
  - Tobu 2y ago
    fclones for example covers it well for any filesystem with reflinks: https://lib.rs/crates/fclones https://lib.rs/crates/fclones fclones group |fclones dedupe
ksec 2y ago
Yes I sort of skip-read a lot of it after that.
the8472 2y ago
ZFS online deduplication is not comparable with on-demand dedup offered btrfs and xfs and is prohibitively expensive for many workloads. The new block cloning still had data corruption bugs quite recently.
- BSDobelix 2y ago
  >ZFS online deduplication is not comparable with on-demand dedup offered btrfs and xfs But it has de-duplication, with your logic no non-CoW FS should be in that list because they are not comparable.

xxmarkuski 2y ago

I'm running bcache, with lvm/luks and xfs on top, since >5 years on my desktop and it has been stable and partition manipulations, like resizes, worked without problems, albeit the tooling is not so well supported. I bought new a new ssd and hdd for my desktop this year and looked into running bcachefs because it offers caching as well as native encryption and cow. I also determined that it is not production ready yet for my use case, my file system is the last thing I want to beta tester of. Investigated using bcache again, but opted to use lvm caching, as it offers better tooling and saves on one layer of block devices (with luks and btrfs on top). Performance is great and partition manipulations also worked flawless. Hopefully bcachefs gains more traction and will be ready for production use, as it combines several useful features. My current setup still feels like making compromises.

NKosmatos 2y ago

Mandatory xkcd comic: https://xkcd.com/927 https://xkcd.com/927 (replace "standards" with "FS")

tjoff 2y ago
Situation is different though, we have very few modern filesystems and have desperately needed some diversion and competition in this area. I've been waiting decades for something like bcachefs, thought it would be btrfs but that turned into a disappointment - for my needs.
- leetnewb 2y ago
  It seems like "modern filesystem" development focus shifted to distributed a few years back.
_flux 2y ago
The key difference is that we don't need to agree on a certain FS, whereas the reason for standards is interoperability. I have both bcachefs and ext4 filesystems on the same machine, for different uses.
jeltz 2y ago
There are only two competitors to bcachefs: btrfs and zfs. So having a third player in this space is a good thing, especially since a lot of people (in my opinion for good reason) do not trust btrfs meaning there is only really zfs.
- PlutoIsAPlanet 2y ago
  ZFS isn't a real competitor given it's not in the kernel and has legal troubles.
  - ZhongXina 2y ago
    Who is downvoting this? Among the large Linux distributions, ZFS is only really supported by Ubuntu, and even that is on the level "Canonical lawyers reviewed this and believe they're safe". If the unmentionable company ever goes to court against them, you're in hot water. You'll have to migrate to FreeBSD or support yourself by building dkms modules. So you're taking a non-zero risk by adopting ZFS. If you're really conservative with these things, as some of us are, you currently don't really have a single safe COW pick. (Smug FreeBSD users incoming.) I have most trust in bcachefs over the long term.

guenthert 2y ago

Hmmh, under "Why bcachefs?" we find - Stability but also - Constant refactorings and later "Disclaimer, my personal data is stored on ZFS" A bit troubling, I find "RAID0 behavior is default when using multiple disks" never have I ever had the need for RAID0 or have I seen a customer using it. I think it was at one time popular with gamers before SSDs became popular and cheap. "RAID 5/6 (experimental) This is referred to as erasure coding and is listed as “DO NOT USE YET”, " Well, you got to start somewhere, but a comparison with btrfs and ZFS seems premature.

GrayShade 2y ago
Does it? From the btrfs docs: > The RAID56 feature provides striping and parity over several devices, same as the traditional RAID5/6. There are some implementation and design deficiencies that make it unreliable for some corner cases and the feature should not be used in production, only for evaluation or testing. The power failure safety for metadata with RAID56 is not 100%.
nextaccountic 2y ago
> "Disclaimer, my personal data is stored on ZFS" > A bit troubling, I find I appreciated the candor The approach of bcachefs developers is that they will only recommend it's usage if it's absolutely, 100% stable and won't eat your data. Bcachefs isn't in that state yet and the developers don't pretend it is. This avoids the kind of trust issues that btrfs has

Liftyee 2y ago

An interesting analysis. I can't stop my brain from parsing the title as "B C A Chefs".

ajb 2y ago

Bcachefs was merged into the kernel only months ago, and had an immediate flurry of bug fixes due to the additional testing this brought. (It was in development for some years before that out of tree). That's the level of maturity that it is at. I think there's a hope that it will become more trustworthy than btrfs due to the developers success with bcache.

snapplebobapple 2y ago
I've been running bcachefs on my main desktop and laptop (My really important data is on my fileserver or in my private git repo (including dotfiles), I'm not crazy) since cachyos made it an install option and it's honestly worked better and caused less problems than btrfs has for me in the past so far. Maybe I was just unlucky but btrfs caused read only filesystem issues and a catastrophic loss on a couple of my computers a few years ago. I am pretty impressed with bcachefs so far.
- bscphil 2y ago
  > since cachyos made it an install option and it's honestly worked better and caused less problems 0 problems in 2.5 months is not necessarily better than 1-2 problems in ~3 years, though. If we're just talking about the single partition boot drive use case, I think I'd go with the option that's had vastly more time to find and eliminate bugs. (If you're conservative about this stuff that probably means ext4, actually.)
  - snapplebobapple 2y ago
    I would agree with you if I didn't run my stuff the way I do (and I'd use zfs or maybe ext4). Pretty much all my important stuff is on a raidz6 file server with a secondary backup raidz6 file server locally pulling backups each night and backups being sent offsite streaming throughout the day. My dotfiles are synced to my local private git repo via yadm (although if I didn't have this system running already I would probably take the time to figure out nix home manager instead of yadm right now). I have a bunch of bash scripts I wrote to automate the most annoying parts of reinstalling as well, so what I am really risking by running bcachefs is about a half hour to reinstall cachyos on whatever system eats it and possibly some minor configuration changes I may not have synced to git via yadm yet.

frankjr 2y ago

> btrfs Encryption Y btrfs doesn't have a built-in encryption. > ZFS Encryption Y I cannot find the discussion right now but I remember reading that they were considering a warning when enabling encryption because it was not really stable and people were running into crashes. https://github.com/openzfs/zfs/issues?q=is%3Aissue+label%3A%22Component%3A+Encryption%22+is%3Aopen+label%3A%22Type%3A+Defect%22 https://github.com/openzfs/zfs/issues?q=is%3Aissue+label%3A%...

prmoustache 2y ago
That bug is old, is missing information and hasn't been closed while the reporter say the problem has been solved after an update. I see it more as an administrative problem than an issue with ZFS encryption.
- GrayShade 2y ago
  See https://gist.github.com/rincebrain/622ee4991732774037ff44c6768085ab#encryption https://gist.github.com/rincebrain/622ee4991732774037ff44c67... though.
nialv7 2y ago
Anecdote: I've been using ZFS encryption for a looong time and never had any problems.

sevg 2y ago

I recently tried btrfs on a new USB thumb drive. I immediately got hard freezes of my main (Linux) OS while working with the USB stick. Never again. I eagerly await bcachefs reaching maturity!

viraptor 2y ago
I hope you reported the issue. That smells like a bug beyond the scope of btrfs itself. The basic filesystem has been stable for a very long time.
nextaccountic 2y ago
Maybe that was a bad USB port? (I have one such port that intermittently disconnects) I have a USB stick with btrfs + LUKS on Arch Linux and it never had a problem like this
- sevg 2y ago
  Same port and USB stick worked fine with XFS and ext4. Tried again with btrfs and hard freezes again.
  - Rinzler89 2y ago
    >Same port and USB stick worked fine with XFS and ext4. None of those file systems are not comparable to BTRFS since they're not COW. BTRFS isn't for crappy USB drives since it has a lot more overhead than EXT4 and XFS which the controllers and flash chips in junky USB drives can't handle.
    - sevg 2y ago
      It's not a crappy USB drive. It's a high-end high-performance Sandisk USB drive. Regardless, I still expect the choice of filesystem to not hard freeze my OS. I've had janky crappy USB drives before and with any other filesystem reads/writes might fail but I don't get a hard freeze. One could argue it could be a bug in the Linux USB stack rather than a bug in the btrfs kernel driver. But what I do know is that when I pick btrfs I get problems.

amluto 2y ago

Here’s my pet peeve regarding RAID: no RAID system I’ve ever used gracefully handles disks that come and go. Concretely: start with two disks in RAID1. Remove one. Mount in degraded mode. Write to a file. Unmount. Reconnect the removed disk. Mount again with both disks. The results vary between annoying (need to restore / “resilver” and have no redundancy until it’s done; massively increased risk of data loss while doing so due to heavy IO load without redundancy and pointless loss of the redundancy that already exists) to catastrophic (outright corruption). The corollary is that RAID invariably works poorly with disks connected over using an interface that enumerates slowly or unreliably. Yet most competent active-active database systems have no problems with this scenario! I would love to see a RAID system that thinks of disks as nodes, properly elects leaders, and can efficiently fast-forward a disk that’s behind. A pile of USB-connected drives would work perfectly, would come up when a quorum was reached, and would behave correctly when only a varying subset of disks is available. Bonus points for also being able to run an array that spans multiple computers efficiently, but that would just be icing on the cake.

TheDong 2y ago
Does ceph not fulfill your requirements here? Especially that last "spans multiple computers" bit.
- amluto 2y ago
  Ceph doesn’t really nail the “I want to boot off this thing” use case. It would be interesting to try, though.
  - magicalhippo 2y ago
    Ceph provides S3-compatible object store no? If so, just use s3backer[1] with a loopback mount and boot[2] off it? I mean, sounds like a house of cards but, should be possible? [1]: https://github.com/archiecobbs/s3backer https://github.com/archiecobbs/s3backer [2]: https://ersei.net/en/blog/fuse-root https://ersei.net/en/blog/fuse-root

curt15 2y ago

How well does bcachefs handle databases and VMs? Those workloads are well-known to be btrfs' kryptonite whereas ZFS seems to tolerate them pretty well as long as one sets the correct recordsize (example: https://www.enterprisedb.com/blog/postgres-vs-file-systems-performance-comparison https://www.enterprisedb.com/blog/postgres-vs-file-systems-p...).

yarg 2y ago
https://www.phoronix.com/review/bcachefs-benchmarks-linux67 https://www.phoronix.com/review/bcachefs-benchmarks-linux67
- dralley 2y ago
  It's worth mentioning that bcachefs has gotten a fair bit of performance work since the last round of Phoronix benchmarks. Also there's some bug where the formatting tool selected 512 byte blocks by default instead of 4k byte blocks on drives where other filesystems picked 4k bytes, which impacted the Phoronix benchmarks. Unsure if that has been fixed yet.
mastax 2y ago
IIRC bcachefs is going to add a non-COW mode which should be good for databases.
- curt15 2y ago
  NoCOW on btrfs is a kludge because it disables the core features of btrfs and is dangerous to use with raid1. Since, ZFS doesn't even have a nocow mode, surely there are other ways of dealing with databases?

linsomniac 2y ago

I had really high hopes for HAMMER2, including that it would one day be ported to Linux, but it seems to have remained firmly planted in Dragonfly BSD and it's not really clear what the status is. https://en.wikipedia.org/wiki/HAMMER2 https://en.wikipedia.org/wiki/HAMMER2

gigatexal 2y ago

> ZFS, pioneering COW filesystem, ... commendable, its block-based design diverges from modern extent-based systems due to complexities in implementing extents with snapshots. why is this a bad thing?

jcalvinowens 2y ago

The idea that a brand new filesystem might be more reliable than good 'ol BTRFS, which Facebook runs on basically their entire infrastructure, is downright laughable to me. Btrfs is also far more reliable than ZFS in my view, because it has far far more real world testing, and is also much more actively developed. Magical perfect elegant code isn't what makes a good filesystem: real world testing, iteration, and bugfixing is. BTRFS has more of that right now than anything else ever has.

jeltz 2y ago
Why is that laughable? I do not think that it is more reliable than btrfs but it is not a crazy idea either. There are a whole bunch of people in these comments with very recent btrfs reliability issues which have affected them and nobody with recent zfs reliability issues.
- jcalvinowens 2y ago
  If anecdotes are meaningful (dubious, but I'll play along...), I can counter with mine: I've been running btrfs on bleeding edge kernels for a decade, and I've never seen a single data loss event. ZFS has corruption bugs, this one was far worse than anything I've seen in btrfs recently: https://lists.freebsd.org/archives/freebsd-stable/2023-November/001726.html https://lists.freebsd.org/archives/freebsd-stable/2023-Novem...
  - tiberious726 2y ago
    I've been running it on hundreds of servers in prod since once of the lead devs gave a talk at LinuxCon 2014 saying it was good to go. Had a few performance issues here and there, especially on older kernels, but never any data loss
  - bjoli 2y ago
    I also think you can't really compare them: ZFS more or less says "never use without ECC memory". BTRFS is run on just about any potato there is. I myself would never run a file server without ECC and a UPS configured for a graceful shutdown. I have also never had any issues, but I only have about 10tb of data.

Tobu 2y ago

> Error handling on CRC read error > 2 or more copies of file, CRC on error, read other copy, data returned to userspace, does not correct bad copy That's been implemented; in Linux 6.11 bcachefs will correct errors on read. See > - Self healing on read IO/checksum error in https://lore.kernel.org/linux-bcachefs/73rweeabpoypzqwyxa7hld7tnkskkaotuo3jjfxnpgn6gg47ly@admkywnz4fsp/ https://lore.kernel.org/linux-bcachefs/73rweeabpoypzqwyxa7hl... Making it possible to scrub from userspace by walking and reading everything (tar -c /mnt/bcachefs >/dev/null).

amtadt 2y ago
Self healing is dangerous because it can potentially corrupt good data on disk, if RAM or other system component is flaky. Repro: supposedly only good copy is copied to ram, ram corrupts bit, crc is recalculated using corrupted but, corrupted copy is written back to disk(s).
- cesarb 2y ago
  > crc is recalculated using corrupted bit Why would it need to recalculate the CRC? The correct CRC (or other hash) for the data is already stored in the metadata trees; it's how it discovered that the data was corrupted in the first place. If it writes back corrupted data, it will be detected as corrupted again the next time.
  - amtadt 2y ago
    Because CRC is in the on-disk data structure, not in the in-ram data structure. It is stripped upon reading to ram, and created upon writing to disk. That's how bcachefs is designed right now.
    - koverstreet 2y ago
      No, we carefully carry around existing checksums when moving data. Page cache is a different story, but doesn't apply to what we're talking about here.
- 2y ago

whalesalad 2y ago

Had to do a double-take on the UI of this blog. It looks identical to my notetaking app, Trilium.

tripdout 2y ago

Does it allow both shrinking and growing the FS? Really wish ZFS allowed shrinking.

commandersaki 2y ago

I’m pretty keen to try bcachefs. Has anyone successfully set it up as root filesystem on a raspberry pi?