3 ms·

> In practice being a true extent-based filesystem with compression and snapshotting is one of bcachefs's distinguishing features (snapshotting is rather obviou

> In practice being a true extent-based filesystem with compression and snapshotting is one of bcachefs's distinguishing features (snapshotting is rather obvious in a block-based system by comparison). Other systems work similarly at a high level but the details are of course quite different in practice. How do bcachefs' extents differ from ZFS's logical blocks (whose size are controlled by the dataset's recordsize property) and how do bcachefs' blocks differ from ZFS's physical blocks (whose size granularity are controlled by the ZFS pool's ashift property)?

aseipp 3y ago
My understanding (Kent is in this thread so maybe I'm totally wrong and he can improve on this) is that extents work like any other filesystem but taken to the extreme: they record a variable, contiguous range of bytes, and that's really all there is in bcachefs, there is no other fixed size unit of data for file data to fit into (I suppose beyond whatever is enforced by the underlying drive geometry and btree index); while a ZFS logical block has several underlying physical blocks of smaller size, but ultimately the logical block is the "unit of operation." It's kind of like the difference between your data type being single scalar number and an interval, I think. bcachefs only has extents and all data is recorded as a variable-sized extent. In contrast, ZFS logical blocks have a fixed size, but may be handled "sparsely" by an underlying number of (smaller) physical blocks. So a 128KiB logical block doesn't mean every 4KiB file takes up 128KiB, naturally. But to change 'recordsize' in ZFS, you also need to rewrite the file, since the logical block size won't be retroactively changed. For good reason, of course. But the indirection between physical/logical makes a big difference in the underlying mechanics. Snapshots make a lot of sense in a system that uses logical, fixed-sized blocks as the atomic "unit" of data even if there is an underlying physical block layer. For example, COW is trivial -- page is read, written and marked dirty, then written out to a new copy of the block it came from. The original logical block can be left alone, and the cost is understood as proportional to the underlying number of physical blocks rewritten (and some change to update metadata). But how does COW or snapshotting work in an extent system? In a non-COW system, you can update the data inside the extent in-place since you don't copy, but in a COW system you need to copy or potentially split the extent which could be costly. You may actually want to group extents together if they're for multiple versions of a file when doing COW. There is no layer of "indirection" above this, so that's where a lot of the trickiness comes from, I think. If you have indirect "logical pointers" instead, you can defer a lot of work, things like reclamation, fragmentation, splitting and merging, etc. There are some other interesting design points in the notes[1] I was looking over to refresh myself, but I don't know how some of them relate to ZFS specifically. For example, another thing bcachefs is pretty good at is low tail latency, and the note on eschewing heavyweight transactions/locks is really good example of that. If you want to run 'touch file', you don't need to take a transaction lock, create an inode, then create the dirent, and release the lock. You can instead simply create the inode and then create dirent assuming that they have a proper write barrier between them, and deleting a file is the inverse operation. Then the system can never be left in an inconsistent state, it can only result in a few dangling objects in the key-value store, which can be garbage collected later. The low tail latency is also assisted by having really small amounts of metadata for each inode, just barely a 64-byte cache line IIRC; so some really weird pathological cases like `ls` in a directory with 500k dirents is way faster in bcachefs than elsewhere. I also wonder if that also falls out from some of its basic design; it basically looks like a POSIX filesystem layered on top of a fast, ordered b-tree, so there's the storage layer and the actual filesystem semantics on top. The filesystem layer probably doesn't actually need to record that much in-band metadata for a given inode when a lot of the underlying consistency and performance issues are handled elsewhere. But that's just me thinking out loud. Overall, there are lots of differences in the details even if some of the high-level stuff comes across as the same. [1] https://bcachefs.org/Architecture/ https://bcachefs.org/Architecture/
- someplaceguy 3y ago
  Just a couple of minor corrections: A ZFS physical block also records a variable, contiguous range of bytes on-disk, not unlike what you describe with bcachefs extents (although, in ZFS the "contiguous" part can get more complicated due to gang blocks and RAID-Z, I think). That said, if you don't have compression enabled, then in ZFS the physical size of the block usually does correspond to the logical size, regardless of how much data was written to it (except for files smaller than one logical block size). Also, if I remember correctly, a ZFS logical block usually only maps to one ZFS physical block, although it can map to up to 3 physical blocks, each of them having the same data (the actual number depends on how many redundant physical copies the logical block is configured to have on top of the ZFS pool redundancy). But in any case, thank you for your long explanation. Some details still escape me, but I can consult the architecture document you linked if I have more doubts.

aseipp 3y ago

My understanding (Kent is in this thread so maybe I'm totally wrong and he can improve on this) is that extents work like any other filesystem but taken to the extreme: they record a variable, contiguous range of bytes, and that's really all there is in bcachefs, there is no other fixed size unit of data for file data to fit into (I suppose beyond whatever is enforced by the underlying drive geometry and btree index); while a ZFS logical block has several underlying physical blocks of smaller size, but ultimately the logical block is the "unit of operation." It's kind of like the difference between your data type being single scalar number and an interval, I think. bcachefs only has extents and all data is recorded as a variable-sized extent. In contrast, ZFS logical blocks have a fixed size, but may be handled "sparsely" by an underlying number of (smaller) physical blocks. So a 128KiB logical block doesn't mean every 4KiB file takes up 128KiB, naturally. But to change 'recordsize' in ZFS, you also need to rewrite the file, since the logical block size won't be retroactively changed. For good reason, of course. But the indirection between physical/logical makes a big difference in the underlying mechanics. Snapshots make a lot of sense in a system that uses logical, fixed-sized blocks as the atomic "unit" of data even if there is an underlying physical block layer. For example, COW is trivial -- page is read, written and marked dirty, then written out to a new copy of the block it came from. The original logical block can be left alone, and the cost is understood as proportional to the underlying number of physical blocks rewritten (and some change to update metadata). But how does COW or snapshotting work in an extent system? In a non-COW system, you can update the data inside the extent in-place since you don't copy, but in a COW system you need to copy or potentially split the extent which could be costly. You may actually want to group extents together if they're for multiple versions of a file when doing COW. There is no layer of "indirection" above this, so that's where a lot of the trickiness comes from, I think. If you have indirect "logical pointers" instead, you can defer a lot of work, things like reclamation, fragmentation, splitting and merging, etc. There are some other interesting design points in the notes[1] I was looking over to refresh myself, but I don't know how some of them relate to ZFS specifically. For example, another thing bcachefs is pretty good at is low tail latency, and the note on eschewing heavyweight transactions/locks is really good example of that. If you want to run 'touch file', you don't need to take a transaction lock, create an inode, then create the dirent, and release the lock. You can instead simply create the inode and then create dirent assuming that they have a proper write barrier between them, and deleting a file is the inverse operation. Then the system can never be left in an inconsistent state, it can only result in a few dangling objects in the key-value store, which can be garbage collected later. The low tail latency is also assisted by having really small amounts of metadata for each inode, just barely a 64-byte cache line IIRC; so some really weird pathological cases like `ls` in a directory with 500k dirents is way faster in bcachefs than elsewhere. I also wonder if that also falls out from some of its basic design; it basically looks like a POSIX filesystem layered on top of a fast, ordered b-tree, so there's the storage layer and the actual filesystem semantics on top. The filesystem layer probably doesn't actually need to record that much in-band metadata for a given inode when a lot of the underlying consistency and performance issues are handled elsewhere. But that's just me thinking out loud. Overall, there are lots of differences in the details even if some of the high-level stuff comes across as the same. [1] https://bcachefs.org/Architecture/ https://bcachefs.org/Architecture/

someplaceguy 3y ago
Just a couple of minor corrections: A ZFS physical block also records a variable, contiguous range of bytes on-disk, not unlike what you describe with bcachefs extents (although, in ZFS the "contiguous" part can get more complicated due to gang blocks and RAID-Z, I think). That said, if you don't have compression enabled, then in ZFS the physical size of the block usually does correspond to the logical size, regardless of how much data was written to it (except for files smaller than one logical block size). Also, if I remember correctly, a ZFS logical block usually only maps to one ZFS physical block, although it can map to up to 3 physical blocks, each of them having the same data (the actual number depends on how many redundant physical copies the logical block is configured to have on top of the ZFS pool redundancy). But in any case, thank you for your long explanation. Some details still escape me, but I can consult the architecture document you linked if I have more doubts.