4 ms·

IMHO bcachefs has important advantages over ZFS. It is far more flexible. ZFS is really similar to traditional block-based RAID. You can get pretty flexible con

IMHO bcachefs has important advantages over ZFS. It is far more flexible. ZFS is really similar to traditional block-based RAID. You can get pretty flexible configurations but 1. They are largely fixed after creation and 2. They only operate at "dataset" level granularity. bachefs has a really flexible design here where you basically add all of your disks to the storage pool and then you can pick redundancy and performance settings per folder (arbitrary subtrees, not just datasets decided at setup time) or even file. For example you can configure a default of 2 replicas for all data, but for your cache directory set it to 1 replica. If you have an important documents folder you can set that to 3 replicas, or 4.2 erasure coding. Similarly you can tell it to put your cache folder on devices labeled "ssd" but your documents folder should write to "ssd" but then be migrated to "hdd" when they are cold. And again, all of this can be set at any time on any subtree. Not just when you initially set up your disks or create the directories.

soupdiver 3y ago
that actually sounds quite neat
FullyFunctional 3y ago
The #1 point of ZFS is protections again bitrot, ie. checksums on all data. Does bcachefs do this?
- lizknope 3y ago
  https://bcachefs.org/ https://bcachefs.org/ It's literally the second item listed on the main web site Full data and metadata checksumming
  - FullyFunctional 3y ago
    Thanks, that's good news. Unfortunately it seems bcachefs isn't using pools like ZFS but instead, like btrfs, creates file systems directly on a collection of disks. That's a bummer if true.
    - wtallis 3y ago
      What are the use cases where you find micromanaging vdevs worthwhile compared to automatic storage allocation that can be changed and rebalanced later? bcachefs does offer several per-device controls over data placement and redundancy that btrfs lacks; are those still not enough?
      - FullyFunctional 3y ago
        Exactly, I _don't_ want to micromanage devices so I throw them all into a pool (with defined redundancy) and then I allocate file systems and vdevs (for iSCSI) from that. With btrfs (and bcachefs) I have to manually assign devices to file systems and I can't have multiple file systems (well, you can have sub-volumes but you can't avoid a big file system in that collection of devices). Maybe I'm missing something, but the pool abstraction makes this very clean and clear.
        wtallis 3y ago
        > so I throw them all into a pool (with defined redundancy) and then I allocate file systems and vdevs (for iSCSI) from that. That sounds backwards. Don't you have to manually define the layout of the vdevs first in order to establish the redundancy, and then allocate the volumes you use for filesystems or iSCSI? If you just do a `zpool create` and give it a dozen disks and ask for raidz2, you're just creating a single vdev that's a RAID6 over all the drives. There's an extra step compared to the btrfs workflow, but if you're not using that opportunity to micromanage your array layout I don't see why you'd prefer that extra step to exist. > and I can't have multiple file systems (well, you can have sub-volumes but you can't avoid a big file system in that collection of devices). Isn't this a purely cosmetic complaint? With at least btrfs, you don't even have to mount the root volume, you can simply mount the subvolumes directly wherever you want them and pretty much ignore the existence of the root volume except when provisioning more subvolumes from the root. You can pretend that you do have a ZFS-style pool abstraction, but one that's navigable like a filesystem in the Unix tradition instead of requiring non-standard tooling to inspect.
    - accelbred 3y ago
- Tuna-Fish 3y ago
  bcachefs currently has full metadata and data checksumming, but there is no scrub implementation. This will likely be in the works soon now that it has been merged.
jl6 3y ago
If you mix a small SSD and a large HDD into a single bcachefs pool, and set it to 2 replicas, do writes have to succeed on both devices before returning success? I.e. is performance constrained by the slowest device? And what happens when the small SSD fills up? Does it carry on writing to the HDD but with only one replica, or does it report no space left?
- wtallis 3y ago
  Are you trying to ask if bcachefs lets you set up stupid configurations? If you only have two devices and you configure it use replication to store two copies of all your data, then you are unavoidably constrained by the capacity of the smaller device. Whether the smaller device is a hard drive or SSD is irrelevant, because neither copy can be regarded as a discardable cache when they're both necessary to maintain the requested level of redundancy.
  - jl6 3y ago
    I am trying to understand bcachefs by asking about an edge case that might be illustrative. > you are unavoidably constrained by the capacity of the smaller device Sure, so what does bcachefs actually do about it? ENOSPC? Answers to the question on synchronous write behavior also welcome.
    - wtallis 3y ago
      > I am trying to understand bcachefs by asking about an edge case that might be illustrative. It looks more like you're questioning whether the filesystem that Linus just merged does obviously wrong things for the simplest test cases of its headline features. Has something given you cause to suspect that this filesystem is so deeply and thoroughly flawed? Because this doesn't quite look like trying to learn, it looks like trying to find an excuse to dismiss bcachefs before even reading any of the docs. Asking if everything about the filesystem is a lie seems really odd.
      - hulitu 3y ago

soupdiver 3y ago

that actually sounds quite neat

FullyFunctional 3y ago

The #1 point of ZFS is protections again bitrot, ie. checksums on all data. Does bcachefs do this?

lizknope 3y ago
https://bcachefs.org/ https://bcachefs.org/ It's literally the second item listed on the main web site Full data and metadata checksumming
- FullyFunctional 3y ago
  Thanks, that's good news. Unfortunately it seems bcachefs isn't using pools like ZFS but instead, like btrfs, creates file systems directly on a collection of disks. That's a bummer if true.
  - wtallis 3y ago
    What are the use cases where you find micromanaging vdevs worthwhile compared to automatic storage allocation that can be changed and rebalanced later? bcachefs does offer several per-device controls over data placement and redundancy that btrfs lacks; are those still not enough?
    - FullyFunctional 3y ago
      Exactly, I _don't_ want to micromanage devices so I throw them all into a pool (with defined redundancy) and then I allocate file systems and vdevs (for iSCSI) from that. With btrfs (and bcachefs) I have to manually assign devices to file systems and I can't have multiple file systems (well, you can have sub-volumes but you can't avoid a big file system in that collection of devices). Maybe I'm missing something, but the pool abstraction makes this very clean and clear.
      - wtallis 3y ago
        > so I throw them all into a pool (with defined redundancy) and then I allocate file systems and vdevs (for iSCSI) from that. That sounds backwards. Don't you have to manually define the layout of the vdevs first in order to establish the redundancy, and then allocate the volumes you use for filesystems or iSCSI? If you just do a `zpool create` and give it a dozen disks and ask for raidz2, you're just creating a single vdev that's a RAID6 over all the drives. There's an extra step compared to the btrfs workflow, but if you're not using that opportunity to micromanage your array layout I don't see why you'd prefer that extra step to exist. > and I can't have multiple file systems (well, you can have sub-volumes but you can't avoid a big file system in that collection of devices). Isn't this a purely cosmetic complaint? With at least btrfs, you don't even have to mount the root volume, you can simply mount the subvolumes directly wherever you want them and pretty much ignore the existence of the root volume except when provisioning more subvolumes from the root. You can pretend that you do have a ZFS-style pool abstraction, but one that's navigable like a filesystem in the Unix tradition instead of requiring non-standard tooling to inspect.
  - accelbred 3y ago
Tuna-Fish 3y ago
bcachefs currently has full metadata and data checksumming, but there is no scrub implementation. This will likely be in the works soon now that it has been merged.

jl6 3y ago

If you mix a small SSD and a large HDD into a single bcachefs pool, and set it to 2 replicas, do writes have to succeed on both devices before returning success? I.e. is performance constrained by the slowest device? And what happens when the small SSD fills up? Does it carry on writing to the HDD but with only one replica, or does it report no space left?

wtallis 3y ago
Are you trying to ask if bcachefs lets you set up stupid configurations? If you only have two devices and you configure it use replication to store two copies of all your data, then you are unavoidably constrained by the capacity of the smaller device. Whether the smaller device is a hard drive or SSD is irrelevant, because neither copy can be regarded as a discardable cache when they're both necessary to maintain the requested level of redundancy.
- jl6 3y ago
  I am trying to understand bcachefs by asking about an edge case that might be illustrative. > you are unavoidably constrained by the capacity of the smaller device Sure, so what does bcachefs actually do about it? ENOSPC? Answers to the question on synchronous write behavior also welcome.
  - wtallis 3y ago
    > I am trying to understand bcachefs by asking about an edge case that might be illustrative. It looks more like you're questioning whether the filesystem that Linus just merged does obviously wrong things for the simplest test cases of its headline features. Has something given you cause to suspect that this filesystem is so deeply and thoroughly flawed? Because this doesn't quite look like trying to learn, it looks like trying to find an excuse to dismiss bcachefs before even reading any of the docs. Asking if everything about the filesystem is a lie seems really odd.
    - hulitu 3y ago