19 ms·

Bcachefs – A New COW Filesystem

292 points by jlpcsl 3y ago

voxadam 3y ago
You're interested in more detailed information about bcachefs I highly recommend checking out bcachefs: Principles of Operation.[0] Also, the original developer of bcachefs (as well as bcache), Kent Overstreet posts status updates from time to time on his Patreon page.[1] [0] https://bcachefs.org/bcachefs-principles-of-operation.pdf https://bcachefs.org/bcachefs-principles-of-operation.pdf [1] https://www.patreon.com/bcachefs https://www.patreon.com/bcachefs
- AceJohnny2 3y ago
  Thanks for the links! I was wondering if bcachefs is architectured with NAND-flash SSD hardware in mind (as recently highlighted on HN in the "Is Sequential IO Dead In The Era Of The NVMe Drive" article [1] [2]), to optimize IO and hardware lifecycle. Skimming through the "bcachefs: Principles Of Operation" PDF, it appears the answer is no. [1] https://jack-vanlightly.com/blog/2023/5/9/is-sequential-io-dead-in-the-era-of-the-nvme-drive https://jack-vanlightly.com/blog/2023/5/9/is-sequential-io-d... [2] https://news.ycombinator.com/item?id=35878961 https://news.ycombinator.com/item?id=35878961
  - koverstreet 3y ago
    It is. There's also plans for ZNS SSD support.
    - LtdJorge 3y ago
      I would listen to what this guy has to say. It looks like he knows something.
- dontlaugh 3y ago
  Interesting that internally it’s much closer to a database than file systems generally are.
orra 3y ago
Oh! This is very exciting. Bcachefs could be the next gen filesystem that Linux needs[1]. Advantages over other filesystems: * ext4 or xfs — these two don't use ECC to protect your data, only the filesystem metadata * zfs — zfs is technically great, but binary distribution of the zfs code is tricky, because the CDDL is GPL incompatible * btrfs — btrfs still doesn't have reliable RAID5 [1] It's been in development for a number of years. It now being proposed for inclusion in the mainline kernel is a major milestone.
- sedatk 3y ago
  > btrfs still doesn't have reliable RAID5 Synology offers btrfs + RAID5 without warning the user. I wonder why they’re so confident with it.
  - bestham 3y ago
    They are running brtrfs on top of DM. https://kb.synology.com/en-nz/DSM/tutorial/What_was_the_RAID_implementation_for_Btrfs_File_System_on_SynologyNAS https://kb.synology.com/en-nz/DSM/tutorial/What_was_the_RAID...
    - sedatk 3y ago
      Thanks for the link!
  - sporkle-feet 3y ago
    Synology doesn't use the btrfs raid - AIUI they layer non-raid btrfs over raid LVM
    - sedatk 3y ago
      Thanks, I wonder what the advantage of using FS-based RAID is.
- vladvasiliu 3y ago
  > * zfs — zfs is technically great, but binary distribution of the zfs code is tricky, because the CDDL is GPL incompatible Building your own ZFS module is easy enough, for example on Arch with zfs-dkms. But there's also the issue of compatibility. Sometimes kernel updates will break ZFS. Even minor ones, 6.2.13 IIRC broke it, whereas 6.2.12 was fine. Right now, 6.3 seems to introduce major compatibility problems. --- edit: looking through the openzfs issues, I was likely thinking of 6.2.8 breaking it, where 6.2.7 was fine. Point stands, though. https://github.com/openzfs/zfs/issues/14658 https://github.com/openzfs/zfs/issues/14658 Regarding 6.3 support, it apparently is merged in the master branch, but no release as of yet. https://github.com/openzfs/zfs/issues/14622 https://github.com/openzfs/zfs/issues/14622
baobrien 3y ago
huh, this is fun: https://lore.kernel.org/lkml/ZFrBEsjrfseCUzqV@moria.home.lan/ https://lore.kernel.org/lkml/ZFrBEsjrfseCUzqV@moria.home.lan... There's a little x86-64 code generator in bcachefs to generate some sort of btree unpacking code.
- dathinab 3y ago
  This is also the point which is the most likely to cause problems for this patch series (which is only fixes and utils added to the kernel) and the bcachefs in general. Like when you have an entry like "bring back function which could make developing viruses easier (through not a vulnerability by themself) related to memory management and code execution" the default answer is nop .. nooop .. never. (Which doesn't mean that it won't come back). It seems while it's not necessary to have this it's a non-neglible performance difference.
  - viraptor 3y ago
    It would be really nice if he posted the difference with/without the optimisation for context. I hope it's going to be included in the explanation post he's planning.
    - kzrdude 3y ago
      It looks like the code generator is only available for x86 anyway, so it seems niche that way. I am all about baseline being good performance, not the special case.
      - BenjiWiebe 3y ago
        He mentions he wants to make the same type of optimization for ARM, so ARM+x86 certainly wouldn't be niche. I wouldn't even call x86 alone niche...
        kzrdude 3y ago
        With eyes on portability, any single-arch-specific code is niche
brnt 3y ago
Erasure coding at the filesystem level? Finally! I've not dared try bcachefs out though, I'm quite wary of data loss, even on my laptop. Does anyone have experience to share?
- BlackLotus89 3y ago
  Had(have) a Laptop that crashed reproducible when touching it wrong. Had a few btrfs corruptions on it and after a while got enough. Have it running bcachefs as rootfs for a few years now and had no issue whatsoever with it. Home is still btrfs (for reasons) and had no data loss on that either. Only problems I had were fixed through booting and mounting it through a rescue system (no fsck necessary) had that twice in 2 years or so. Was too lazy to check what the bcachefs hook (aur package) does wrong. Edit: Reasons for home being btrfs. I set this up a long fucking time ago and it was more or less meant as a stresstest for bcachefs. Since I didn't want data loss on important data (like my home) I left my home as btrfs
sporkle-feet 3y ago
The feature that caught my eye is the concept of having different targets. A fast SSD can be set as the target for foreground writes, but that data will be transparently copied in the background to a "background" target, i.e. a large/slow disk. If this works, it will be awesome.
- viraptor 3y ago
  You can also have that at block level (which is where bcache itself comes from). Facebook used it years ago and I had it on an SSD+HDD laptop... a decade ago at least? Unless you want the filesystem to know about it, it's ready to go now.
  - jwilk 3y ago
    Look up --write-mostly and --write-behind options in mdadm(8) man page. I can't recommend such a setup though. It works very poorly for me.
    - saltcured 3y ago
      See the lvmcache(7) manpage, which I think may be what the earlier poster was thinking of. It isn't an asymmetric RAID mode, but a tiered caching scheme where you can, for example, put a faster and smaller enterprise SSD in front of a larger and slower bulk store. So you can have a large bulk volume but the recently/frequently used blocks get the performance of the fast cache volume. I set it up in the past with an mdadm RAID1 array over SSDs as a caching layer in front of another mdadm array over HDDs. It performed quite well in a developer/compute workstation environment.
      - viraptor 3y ago
        I did mean bcache specifically. https://www.kernel.org/doc/Documentation/bcache.txt https://www.kernel.org/doc/Documentation/bcache.txt
- throw0101a 3y ago
eis 3y ago
I really hope Linux can get a modern FS into common usage (as in default FS for most distros). After more than a decade ZFS and BTRFS didn't go anywhere. Something that's just there as a default, is stable, performs decently (at least on ext4 level) and brings modern features like snapshots. Bcachefs seems to have a decent shot. What I'd like to see even more though would be a switch from the existing posix based filesystem APIs to a transaction based system. It is way too complicated to do filesystem operations that are not prone to data corruption should there be any issues.
- viraptor 3y ago
  Btrfs is the default on a few systems already like Fedora, suse, Garuda, easynas, rockstor, and some others. It's not the default in Ubuntu and Debian, but I wouldn't say it didn't go anywhere either.
  - jadbox 3y ago
    I'm using BTRfs on Fedora (by default install) and it's been great over the last year. The only thing to be aware of is to disable CoW/Hashing on database stores or streaming download folders. Otherwise it'll rehash each file update, which isn't needed.
    - dnzm 3y ago
      I've been running it on my NAS-slash-homeserver for... 5 or 6 years now, I think. Root on a single SSD, data on a few HDDs in RAID1. It's been great so far. My desktops are all btrfs too, and the integration between OpenSUSE's package manager and btrfs snapshots has been useful more than once.
    - giantrobot 3y ago
      Why is it hashing files and not blocks? If a block is hashed and written there's no need to touch it again.
      - 5e92cb50239222b 3y ago
harvie 3y ago
What i really miss when compared to ZFS is ability to create datasets. I really like to use ZFS subvolumes for LXC containers. That way i can have separate sub-btree for each container with it's own size limit without having to create partitions or LVs, format the filesystem and then resize everything when i need to grow the partition or even defragment fs before shrinking it. With ZFS i can easily give and take disk capacity to my containers without having to do any multi step operation that requires close attention to prevent accidental data loss. Basicaly i just state what size i want that subtree to be and it happens without having to touch underlying block devices. Also i can change it anytime during runtime extremely easily. Eg.: zfs set quota=42G tank/vps/my_vps zfs set quota=32G tank/vps/my_vps zfs set quota=23G tank/vps/my_other_vps btrfs can kinda do this as well, but the commands are not as straighforward as in zfs. update: My bad. bcachefs seems to have subvolumes now. there is also some quota support, but so far the documentation is bit lacking, so not yet sure how to use that and if that can be configured per dataset.
curt15 3y ago
For some reason VM and DB workloads are btrfs's Achilles heel but ZFS seems to handle them pretty well (provided that a suitable recordsize is set). How do they perform on bcachefs?
- candiddevmike 3y ago
  I've never had a problem with these on BTRFS with COW disabled on their directories...
  - _piif 3y ago
    The issue is that also disables many of the interesting features of BTRFS for those files. No checksumming, no snapshots and no compression. In comparison ZFS handles these features just fine for those kinds of files without the enormous performance / fragmentation issues of BTRFS (without nodatacow).
    - yencabulator 3y ago
      chattr +C does not disable snapshots. It switches the writes from CoW-everything to CoW-only-after-snapshot. Snapshots work just fine, with only the first write (per extent) after a snapshot paying the CoW cost.
      - _piif 3y ago
        I was not aware this was the case, TIL!
    - deleted 3y ago
      [deleted]
graderjs 3y ago
Is there an optimal filesystem or is it all just trade offs? And how far if we come since you know when we were first creating file systems plan nine or whatever to now like it’s as there been any sort of technological Leap like killer algorithm it’s really improve things?
- the8472 3y ago
  It's all tradeoffs.
- dsr_ 3y ago
  An optimal filesystem... for what? There is no single filesystem which is optimized for everything, so you need to specify things like cross-platform transportability, network transparency, hardware interfaces, hardware capability, reliability requirements, cost-effectiveness, required features, expected workload, licensing and what the track record is in the real world.
throw0101b 3y ago
> These are RW btrfs-style snapshots There's a word for 'RW snapshots': clones. E.g. * https://docs.netapp.com/us-en/ontap/task_admin_clone_data.html https://docs.netapp.com/us-en/ontap/task_admin_clone_data.ht... * http://doc.isilon.com/onefs/9.4.0/help/en-us/ifs_t_clone_a_file_from_a_snapshot.html http://doc.isilon.com/onefs/9.4.0/help/en-us/ifs_t_clone_a_f... * https://openzfs.github.io/openzfs-docs/man/8/zfs-clone.8.html https://openzfs.github.io/openzfs-docs/man/8/zfs-clone.8.htm... * http://www.voleg.info/lvm2-clone-logical-volume.html http://www.voleg.info/lvm2-clone-logical-volume.html In every other implementation I've come across the word "snapshot" is about read-only copies. I'm not sure why btrfs (and now bcachefs?) thinks it needs to muddy the nomenclature waters.
- webstrand 3y ago
  Cloning can also mean simple duplication. I think calling it a RW snapshot is clearer because a snapshot generally doesn't mean simple duplication.
  - throw0101a 3y ago
    > I think calling it a RW snapshot […] So what do you call a RO snapshot? Or do you now need to write the prefix "RO" and "RW" everywhere when referring to a "snapshot"? How do CLI commands work? Will you have "btrfs snapshot" and then have to always define whether you want RO or RW on every invocation? This smells like git's bad front-end CLI porcelain all over again (regardless of how nice the back-end plumbing may be). This is a solved problem with an established nomenclature IMHO: just use the already-existing nouns/CLI-verbs of "snapshot" and "clone". > […] is clearer because a snapshot generally doesn't mean simple duplication. A snapshot generally means a static copy of the data, with bachefs (and ZFS and btrfs) being CoW then it implies new copies are not needed unless/until the source is altered. If you want deduplication use "dedupe" in your CLI.
    - nextaccountic 3y ago
      > So what do you call a RO snapshot There should be no read-only snapshot: it's just a writable snapshot where you don't happen to perform a write
londons_explore 3y ago
I would like to see filesystems benchmarked for robustness. Specifically, robustness to everything around them not performing as required. For example, imagine an evil SSD which had a 1% chance of rolling a sector back to a previous version, a 1% chance of saying a write failed when it didn't, a 1% chance of writing data to the wrong sector number, a 1% chance of flipping some bits in the written data, and a 1% chance of disconnecting and reconnecting a few seconds later. Real SSD's have bugs that make them do all of these things. Given this evil SSD, I want to know how long the filesystem can keep going serving the users usecase.
- colttt 3y ago
  Kent use md-faulty to test such a scenario, you can also see here the test-CI https://evilpiepirate.org/~testdashboard/ci https://evilpiepirate.org/~testdashboard/ci
- 112233 3y ago
  Please, where can i read more about this. I remember bricking OCZ drive by setting ATA password, as was fashionable to do back then, but 1% writes going to wrong sector - what are these drives, fake sd cards from aliexpress? Like, which manufacturer goes, like "tests show we cannot write more that 400kB without corrupting drive, let us ship this!" ?
  - londons_explore 3y ago
    Writes going to the wrong sector are usually wear levelling algorithms gone wrong. Specifically, it normally means the information about which logical sector maps to which physical sector was updated not in sync with the actual writing of the data. This is a common performance 'trick' - by delaying and aggregating these bookkeeping writes, and taking them off the critical path, you avoid writing so much data and the user sees lower latency. However, if something like a power failure or firmware crash happens, and the bookkeeping writes never happen, then the end result that the user sees after a reboot is their data written to the wrong sector.
  - jlokier 3y ago
jacknews 3y ago
"The COW filesystem for Linux that won't eat your data" LOL, they know what the problem is at least. I will try it out on some old hard disks. The others (esp. looking at you btrfs) are not good at not losing your entire volumes when disks start to go bad.
IAmLiterallyAB 3y ago
Here's a link to the Bcachefs site https://bcachefs.org/ https://bcachefs.org/ I think it summarizes its features and strengths pretty well, and it has a lot of good technical information.
- sumtechguy 3y ago
  Does anyone know if there any good links to current benchmarks between the diff types? My googlefu is only finding stuff form 2019.
- anentropic 3y ago
  I can't help reading this name as Bca-chefs (...I realise it must be B-cache-fs)
  - p1mrx 3y ago
    Maybe we could call it b$fs
  - Dylan16807 3y ago
    Can you elaborate on how you pronounce "Bca"?
    - chris1993 3y ago
      I was mentally pronouncing it ‘beaker-chefs’
deleted 3y ago
[deleted]
MisterTea 3y ago
Another File system I am interested in is GEFS - good enough fs (rather - "great experimental file shredder" until stable ;-). It's based on B-epsilon trees, a data structure which wasn't around when ZFS was designed. The idea is to build a ZFS like fs without the size and complexity of zfs. So far its plan 9 only and not production ready though there is a chance it could be ported to OpenBSD and a talk was given at NYC*BUG: https://www.nycbug.org/index?action=view&id=10688 https://www.nycbug.org/index?action=view&id=10688 Code: http://shithub.us/ori/gefs/HEAD/info.html http://shithub.us/ori/gefs/HEAD/info.html
renewiltord 3y ago
Is there a high-performance in-kernel FS that acts as a hierarchical cache that I can export over NFS? Presently I use `catfs` over `goofys` and then I export the `catfs` mount.
- cdavid 3y ago
  Not sure I understand your use case, but if you have to use nfs, cachefilesd is very effective for read heavy workload: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_file_systems/getting-started-with-fs-cache_managing-file-systems https://access.redhat.com/documentation/en-us/red_hat_enterp...
  - renewiltord 3y ago
    Ah, I'm hoping for the other way around: slow storage ----> cache ----> export over nfs ---> client
layer8 3y ago
I parsed this as “BCA chefs” at first.
shmerl 3y ago
Cool to see it approaching upstreaming. Does it support mountable subvolumes like btrfs?
unixhero 3y ago
How does this compare with ZFS?

292 points by jlpcsl 3y ago

You're interested in more detailed information about bcachefs I highly recommend checking out bcachefs: Principles of Operation.[0] Also, the original developer of bcachefs (as well as bcache), Kent Overstreet posts status updates from time to time on his Patreon page.[1] [0] https://bcachefs.org/bcachefs-principles-of-operation.pdf https://bcachefs.org/bcachefs-principles-of-operation.pdf [1] https://www.patreon.com/bcachefs https://www.patreon.com/bcachefs

AceJohnny2 3y ago
Thanks for the links! I was wondering if bcachefs is architectured with NAND-flash SSD hardware in mind (as recently highlighted on HN in the "Is Sequential IO Dead In The Era Of The NVMe Drive" article [1] [2]), to optimize IO and hardware lifecycle. Skimming through the "bcachefs: Principles Of Operation" PDF, it appears the answer is no. [1] https://jack-vanlightly.com/blog/2023/5/9/is-sequential-io-dead-in-the-era-of-the-nvme-drive https://jack-vanlightly.com/blog/2023/5/9/is-sequential-io-d... [2] https://news.ycombinator.com/item?id=35878961 https://news.ycombinator.com/item?id=35878961
- koverstreet 3y ago
  It is. There's also plans for ZNS SSD support.
  - LtdJorge 3y ago
    I would listen to what this guy has to say. It looks like he knows something.
dontlaugh 3y ago
Interesting that internally it’s much closer to a database than file systems generally are.

orra 3y ago

Oh! This is very exciting. Bcachefs could be the next gen filesystem that Linux needs[1]. Advantages over other filesystems: * ext4 or xfs — these two don't use ECC to protect your data, only the filesystem metadata * zfs — zfs is technically great, but binary distribution of the zfs code is tricky, because the CDDL is GPL incompatible * btrfs — btrfs still doesn't have reliable RAID5 [1] It's been in development for a number of years. It now being proposed for inclusion in the mainline kernel is a major milestone.

sedatk 3y ago
> btrfs still doesn't have reliable RAID5 Synology offers btrfs + RAID5 without warning the user. I wonder why they’re so confident with it.
- bestham 3y ago
  They are running brtrfs on top of DM. https://kb.synology.com/en-nz/DSM/tutorial/What_was_the_RAID_implementation_for_Btrfs_File_System_on_SynologyNAS https://kb.synology.com/en-nz/DSM/tutorial/What_was_the_RAID...
  - sedatk 3y ago
    Thanks for the link!
- sporkle-feet 3y ago
  Synology doesn't use the btrfs raid - AIUI they layer non-raid btrfs over raid LVM
  - sedatk 3y ago
    Thanks, I wonder what the advantage of using FS-based RAID is.
vladvasiliu 3y ago
> * zfs — zfs is technically great, but binary distribution of the zfs code is tricky, because the CDDL is GPL incompatible Building your own ZFS module is easy enough, for example on Arch with zfs-dkms. But there's also the issue of compatibility. Sometimes kernel updates will break ZFS. Even minor ones, 6.2.13 IIRC broke it, whereas 6.2.12 was fine. Right now, 6.3 seems to introduce major compatibility problems. --- edit: looking through the openzfs issues, I was likely thinking of 6.2.8 breaking it, where 6.2.7 was fine. Point stands, though. https://github.com/openzfs/zfs/issues/14658 https://github.com/openzfs/zfs/issues/14658 Regarding 6.3 support, it apparently is merged in the master branch, but no release as of yet. https://github.com/openzfs/zfs/issues/14622 https://github.com/openzfs/zfs/issues/14622

baobrien 3y ago

huh, this is fun: https://lore.kernel.org/lkml/ZFrBEsjrfseCUzqV@moria.home.lan/ https://lore.kernel.org/lkml/ZFrBEsjrfseCUzqV@moria.home.lan... There's a little x86-64 code generator in bcachefs to generate some sort of btree unpacking code.

dathinab 3y ago
This is also the point which is the most likely to cause problems for this patch series (which is only fixes and utils added to the kernel) and the bcachefs in general. Like when you have an entry like "bring back function which could make developing viruses easier (through not a vulnerability by themself) related to memory management and code execution" the default answer is nop .. nooop .. never. (Which doesn't mean that it won't come back). It seems while it's not necessary to have this it's a non-neglible performance difference.
- viraptor 3y ago
  It would be really nice if he posted the difference with/without the optimisation for context. I hope it's going to be included in the explanation post he's planning.
  - kzrdude 3y ago
    It looks like the code generator is only available for x86 anyway, so it seems niche that way. I am all about baseline being good performance, not the special case.
    - BenjiWiebe 3y ago
      He mentions he wants to make the same type of optimization for ARM, so ARM+x86 certainly wouldn't be niche. I wouldn't even call x86 alone niche...
      - kzrdude 3y ago
        With eyes on portability, any single-arch-specific code is niche

brnt 3y ago

Erasure coding at the filesystem level? Finally! I've not dared try bcachefs out though, I'm quite wary of data loss, even on my laptop. Does anyone have experience to share?

BlackLotus89 3y ago
Had(have) a Laptop that crashed reproducible when touching it wrong. Had a few btrfs corruptions on it and after a while got enough. Have it running bcachefs as rootfs for a few years now and had no issue whatsoever with it. Home is still btrfs (for reasons) and had no data loss on that either. Only problems I had were fixed through booting and mounting it through a rescue system (no fsck necessary) had that twice in 2 years or so. Was too lazy to check what the bcachefs hook (aur package) does wrong. Edit: Reasons for home being btrfs. I set this up a long fucking time ago and it was more or less meant as a stresstest for bcachefs. Since I didn't want data loss on important data (like my home) I left my home as btrfs

sporkle-feet 3y ago

The feature that caught my eye is the concept of having different targets. A fast SSD can be set as the target for foreground writes, but that data will be transparently copied in the background to a "background" target, i.e. a large/slow disk. If this works, it will be awesome.

viraptor 3y ago
You can also have that at block level (which is where bcache itself comes from). Facebook used it years ago and I had it on an SSD+HDD laptop... a decade ago at least? Unless you want the filesystem to know about it, it's ready to go now.
- jwilk 3y ago
  Look up --write-mostly and --write-behind options in mdadm(8) man page. I can't recommend such a setup though. It works very poorly for me.
  - saltcured 3y ago
    See the lvmcache(7) manpage, which I think may be what the earlier poster was thinking of. It isn't an asymmetric RAID mode, but a tiered caching scheme where you can, for example, put a faster and smaller enterprise SSD in front of a larger and slower bulk store. So you can have a large bulk volume but the recently/frequently used blocks get the performance of the fast cache volume. I set it up in the past with an mdadm RAID1 array over SSDs as a caching layer in front of another mdadm array over HDDs. It performed quite well in a developer/compute workstation environment.
    - viraptor 3y ago
      I did mean bcache specifically. https://www.kernel.org/doc/Documentation/bcache.txt https://www.kernel.org/doc/Documentation/bcache.txt
throw0101a 3y ago

eis 3y ago

I really hope Linux can get a modern FS into common usage (as in default FS for most distros). After more than a decade ZFS and BTRFS didn't go anywhere. Something that's just there as a default, is stable, performs decently (at least on ext4 level) and brings modern features like snapshots. Bcachefs seems to have a decent shot. What I'd like to see even more though would be a switch from the existing posix based filesystem APIs to a transaction based system. It is way too complicated to do filesystem operations that are not prone to data corruption should there be any issues.

viraptor 3y ago
Btrfs is the default on a few systems already like Fedora, suse, Garuda, easynas, rockstor, and some others. It's not the default in Ubuntu and Debian, but I wouldn't say it didn't go anywhere either.
- jadbox 3y ago
  I'm using BTRfs on Fedora (by default install) and it's been great over the last year. The only thing to be aware of is to disable CoW/Hashing on database stores or streaming download folders. Otherwise it'll rehash each file update, which isn't needed.
  - dnzm 3y ago
    I've been running it on my NAS-slash-homeserver for... 5 or 6 years now, I think. Root on a single SSD, data on a few HDDs in RAID1. It's been great so far. My desktops are all btrfs too, and the integration between OpenSUSE's package manager and btrfs snapshots has been useful more than once.
  - giantrobot 3y ago
    Why is it hashing files and not blocks? If a block is hashed and written there's no need to touch it again.
    - 5e92cb50239222b 3y ago

harvie 3y ago

What i really miss when compared to ZFS is ability to create datasets. I really like to use ZFS subvolumes for LXC containers. That way i can have separate sub-btree for each container with it's own size limit without having to create partitions or LVs, format the filesystem and then resize everything when i need to grow the partition or even defragment fs before shrinking it. With ZFS i can easily give and take disk capacity to my containers without having to do any multi step operation that requires close attention to prevent accidental data loss. Basicaly i just state what size i want that subtree to be and it happens without having to touch underlying block devices. Also i can change it anytime during runtime extremely easily. Eg.: zfs set quota=42G tank/vps/my_vps zfs set quota=32G tank/vps/my_vps zfs set quota=23G tank/vps/my_other_vps btrfs can kinda do this as well, but the commands are not as straighforward as in zfs. update: My bad. bcachefs seems to have subvolumes now. there is also some quota support, but so far the documentation is bit lacking, so not yet sure how to use that and if that can be configured per dataset.

curt15 3y ago

For some reason VM and DB workloads are btrfs's Achilles heel but ZFS seems to handle them pretty well (provided that a suitable recordsize is set). How do they perform on bcachefs?

candiddevmike 3y ago
I've never had a problem with these on BTRFS with COW disabled on their directories...
- _piif 3y ago
  The issue is that also disables many of the interesting features of BTRFS for those files. No checksumming, no snapshots and no compression. In comparison ZFS handles these features just fine for those kinds of files without the enormous performance / fragmentation issues of BTRFS (without nodatacow).
  - yencabulator 3y ago
    chattr +C does not disable snapshots. It switches the writes from CoW-everything to CoW-only-after-snapshot. Snapshots work just fine, with only the first write (per extent) after a snapshot paying the CoW cost.
    - _piif 3y ago
      I was not aware this was the case, TIL!
  - deleted 3y ago
    [deleted]

graderjs 3y ago

Is there an optimal filesystem or is it all just trade offs? And how far if we come since you know when we were first creating file systems plan nine or whatever to now like it’s as there been any sort of technological Leap like killer algorithm it’s really improve things?

the8472 3y ago
It's all tradeoffs.
dsr_ 3y ago
An optimal filesystem... for what? There is no single filesystem which is optimized for everything, so you need to specify things like cross-platform transportability, network transparency, hardware interfaces, hardware capability, reliability requirements, cost-effectiveness, required features, expected workload, licensing and what the track record is in the real world.

throw0101b 3y ago

> These are RW btrfs-style snapshots There's a word for 'RW snapshots': clones. E.g. * https://docs.netapp.com/us-en/ontap/task_admin_clone_data.html https://docs.netapp.com/us-en/ontap/task_admin_clone_data.ht... * http://doc.isilon.com/onefs/9.4.0/help/en-us/ifs_t_clone_a_file_from_a_snapshot.html http://doc.isilon.com/onefs/9.4.0/help/en-us/ifs_t_clone_a_f... * https://openzfs.github.io/openzfs-docs/man/8/zfs-clone.8.html https://openzfs.github.io/openzfs-docs/man/8/zfs-clone.8.htm... * http://www.voleg.info/lvm2-clone-logical-volume.html http://www.voleg.info/lvm2-clone-logical-volume.html In every other implementation I've come across the word "snapshot" is about read-only copies. I'm not sure why btrfs (and now bcachefs?) thinks it needs to muddy the nomenclature waters.

webstrand 3y ago
Cloning can also mean simple duplication. I think calling it a RW snapshot is clearer because a snapshot generally doesn't mean simple duplication.
- throw0101a 3y ago
  > I think calling it a RW snapshot […] So what do you call a RO snapshot? Or do you now need to write the prefix "RO" and "RW" everywhere when referring to a "snapshot"? How do CLI commands work? Will you have "btrfs snapshot" and then have to always define whether you want RO or RW on every invocation? This smells like git's bad front-end CLI porcelain all over again (regardless of how nice the back-end plumbing may be). This is a solved problem with an established nomenclature IMHO: just use the already-existing nouns/CLI-verbs of "snapshot" and "clone". > […] is clearer because a snapshot generally doesn't mean simple duplication. A snapshot generally means a static copy of the data, with bachefs (and ZFS and btrfs) being CoW then it implies new copies are not needed unless/until the source is altered. If you want deduplication use "dedupe" in your CLI.
  - nextaccountic 3y ago
    > So what do you call a RO snapshot There should be no read-only snapshot: it's just a writable snapshot where you don't happen to perform a write

londons_explore 3y ago

I would like to see filesystems benchmarked for robustness. Specifically, robustness to everything around them not performing as required. For example, imagine an evil SSD which had a 1% chance of rolling a sector back to a previous version, a 1% chance of saying a write failed when it didn't, a 1% chance of writing data to the wrong sector number, a 1% chance of flipping some bits in the written data, and a 1% chance of disconnecting and reconnecting a few seconds later. Real SSD's have bugs that make them do all of these things. Given this evil SSD, I want to know how long the filesystem can keep going serving the users usecase.

colttt 3y ago
Kent use md-faulty to test such a scenario, you can also see here the test-CI https://evilpiepirate.org/~testdashboard/ci https://evilpiepirate.org/~testdashboard/ci
112233 3y ago
Please, where can i read more about this. I remember bricking OCZ drive by setting ATA password, as was fashionable to do back then, but 1% writes going to wrong sector - what are these drives, fake sd cards from aliexpress? Like, which manufacturer goes, like "tests show we cannot write more that 400kB without corrupting drive, let us ship this!" ?
- londons_explore 3y ago
  Writes going to the wrong sector are usually wear levelling algorithms gone wrong. Specifically, it normally means the information about which logical sector maps to which physical sector was updated not in sync with the actual writing of the data. This is a common performance 'trick' - by delaying and aggregating these bookkeeping writes, and taking them off the critical path, you avoid writing so much data and the user sees lower latency. However, if something like a power failure or firmware crash happens, and the bookkeeping writes never happen, then the end result that the user sees after a reboot is their data written to the wrong sector.
- jlokier 3y ago

jacknews 3y ago

"The COW filesystem for Linux that won't eat your data" LOL, they know what the problem is at least. I will try it out on some old hard disks. The others (esp. looking at you btrfs) are not good at not losing your entire volumes when disks start to go bad.

IAmLiterallyAB 3y ago

Here's a link to the Bcachefs site https://bcachefs.org/ https://bcachefs.org/ I think it summarizes its features and strengths pretty well, and it has a lot of good technical information.

sumtechguy 3y ago
Does anyone know if there any good links to current benchmarks between the diff types? My googlefu is only finding stuff form 2019.
anentropic 3y ago
I can't help reading this name as Bca-chefs (...I realise it must be B-cache-fs)
- p1mrx 3y ago
  Maybe we could call it b$fs
- Dylan16807 3y ago
  Can you elaborate on how you pronounce "Bca"?
  - chris1993 3y ago
    I was mentally pronouncing it ‘beaker-chefs’

deleted 3y ago

[deleted]

MisterTea 3y ago

Another File system I am interested in is GEFS - good enough fs (rather - "great experimental file shredder" until stable ;-). It's based on B-epsilon trees, a data structure which wasn't around when ZFS was designed. The idea is to build a ZFS like fs without the size and complexity of zfs. So far its plan 9 only and not production ready though there is a chance it could be ported to OpenBSD and a talk was given at NYC*BUG: https://www.nycbug.org/index?action=view&id=10688 https://www.nycbug.org/index?action=view&id=10688 Code: http://shithub.us/ori/gefs/HEAD/info.html http://shithub.us/ori/gefs/HEAD/info.html

renewiltord 3y ago

Is there a high-performance in-kernel FS that acts as a hierarchical cache that I can export over NFS? Presently I use `catfs` over `goofys` and then I export the `catfs` mount.

cdavid 3y ago
Not sure I understand your use case, but if you have to use nfs, cachefilesd is very effective for read heavy workload: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_file_systems/getting-started-with-fs-cache_managing-file-systems https://access.redhat.com/documentation/en-us/red_hat_enterp...
- renewiltord 3y ago
  Ah, I'm hoping for the other way around: slow storage ----> cache ----> export over nfs ---> client

layer8 3y ago

I parsed this as “BCA chefs” at first.

shmerl 3y ago

Cool to see it approaching upstreaming. Does it support mountable subvolumes like btrfs?

unixhero 3y ago

How does this compare with ZFS?