7 ms·

Reading the peer comment document, the unique thing about compression with bcachefs is that it compresses extents, rather than blocks. Where a compressed and c

by tyingq 3y ago

Reading the peer comment document, the unique thing about compression with bcachefs is that it compresses extents, rather than blocks. Where a compressed and checksummed extent is 64Kb, and a block is typically 2/4/8 Kb. So better compression at the cost of slightly reduced performance for random reads. Edit: I was paraphrasing the text in the peer linked document: "As with data checksumming, we compress entire extents, not individual disk blocks - this gives us better compression ratios than other filesystems, at the cost of reduced small random read performance."

cwillu 3y ago
That's not unique to bcachefs; it's how transparent compression is implemented in every filesystem that implements it that I'm aware of (for example, I just checked NTFS, BTRFS and ZFS).
- mastax 3y ago
  ZFS compresses records (blocks) independently.
  - someplaceguy 3y ago
    A ZFS block is 128 KB by default (smaller if the file is < 128 KB, but can also be larger if you manually configure it). So a ZFS block is even larger than a bcachefs "extent".
    - mastax 3y ago
      Well yes, my point is that (counter to GP) ZFS doesn't combine together multiple blocks into a compression extent, rather it uses larger blocks by default.
      - someplaceguy 3y ago
        ZFS's larger blocks you are referring to are the logical block size, which is the size unit for compression among other things. I don't know much about bcachefs, but as I understand it, a logical block in ZFS is equivalent to a bcachefs extent (in terms of compression, at least) and a ZFS physical block is equivalent to a (possibly empty) set of bcachefs blocks. As an example, in terms of actual on-disk storage, a 128 KB logical block in ZFS can be compressed down to (for example) a 3.5 KB "physical" block and it will only actually consume 3.5 KB of disk space (surprisingly, the block size of a ZFS physical block doesn't have to be a power of 2!). That said, if you have 4 KB-sector disks then you should probably use a physical block size granularity of 4 KB for your ZFS pool rather than 512 bytes. So, regardless of the file size, an on-disk ZFS block can actually occupy anywhere from 0 bytes up to 1 MB, with a granularity of up to 512 bytes, depending on how much it is compressed, what the physical block size granularity of the ZFS pool is and what you've selected as the recordsize property (i.e. the logical block size) for the corresponding filesystem in the pool.
        rincebrain 3y ago
        16M, not 1M.
        someplaceguy 3y ago
        Thanks for the correction. The `zfsprops` man page should be updated to reflect that...
        rincebrain 3y ago
        It has been. The default was to limit it to 1M in case there were weird edge cases with the larger sizes (16M is the largest it'll encode on disk without a format change, IIRC), with larger sizes behind a tunable, but after a few years of people who knew this twiddling it, the default in 2.2 will be that the tunable is set to 16M.
        5e92cb50239222b 3y ago
        > if you have 4 KB-sector disks then you should probably use a physical block size granularity of 4 KB for your ZFS pool rather than 512 bytes According to Allan Jude (OpenZFS & FreeBSD developer), you should never use 512-byte recordsize even on an HDD with 512-byte sectors. Always start from 4 KiBs, probably even from 8. https://klarasystems.com/articles/tuning-recordsize-in-openzfs https://klarasystems.com/articles/tuning-recordsize-in-openz...
- deleted 3y ago
  [deleted]
- aseipp 3y ago
  ZFS is block based, not extent based, but blocks can be variable size from my understanding. Btrfs does compress individual blocks at 4KiB granularity, but can use the same zstd dictionary for up-to 128KiB runs of contiguous blocks. I don't know about modern NTFS but IIRC it can't compress logical blocks larger than 4KiB anyway, or couldn't a long time ago, so at that point it's kind of moot when compared to the greater ratios of the competitors. In practice being a true extent-based filesystem with compression and snapshotting is one of bcachefs's distinguishing features (snapshotting is rather obvious in a block-based system by comparison). Other systems work similarly at a high level but the details are of course quite different in practice.
  - someplaceguy 3y ago
    > In practice being a true extent-based filesystem with compression and snapshotting is one of bcachefs's distinguishing features (snapshotting is rather obvious in a block-based system by comparison). Other systems work similarly at a high level but the details are of course quite different in practice. How do bcachefs' extents differ from ZFS's logical blocks (whose size are controlled by the dataset's recordsize property) and how do bcachefs' blocks differ from ZFS's physical blocks (whose size granularity are controlled by the ZFS pool's ashift property)?
    - aseipp 3y ago
      My understanding (Kent is in this thread so maybe I'm totally wrong and he can improve on this) is that extents work like any other filesystem but taken to the extreme: they record a variable, contiguous range of bytes, and that's really all there is in bcachefs, there is no other fixed size unit of data for file data to fit into (I suppose beyond whatever is enforced by the underlying drive geometry and btree index); while a ZFS logical block has several underlying physical blocks of smaller size, but ultimately the logical block is the "unit of operation." It's kind of like the difference between your data type being single scalar number and an interval, I think. bcachefs only has extents and all data is recorded as a variable-sized extent. In contrast, ZFS logical blocks have a fixed size, but may be handled "sparsely" by an underlying number of (smaller) physical blocks. So a 128KiB logical block doesn't mean every 4KiB file takes up 128KiB, naturally. But to change 'recordsize' in ZFS, you also need to rewrite the file, since the logical block size won't be retroactively changed. For good reason, of course. But the indirection between physical/logical makes a big difference in the underlying mechanics. Snapshots make a lot of sense in a system that uses logical, fixed-sized blocks as the atomic "unit" of data even if there is an underlying physical block layer. For example, COW is trivial -- page is read, written and marked dirty, then written out to a new copy of the block it came from. The original logical block can be left alone, and the cost is understood as proportional to the underlying number of physical blocks rewritten (and some change to update metadata). But how does COW or snapshotting work in an extent system? In a non-COW system, you can update the data inside the extent in-place since you don't copy, but in a COW system you need to copy or potentially split the extent which could be costly. You may actually want to group extents together if they're for multiple versions of a file when doing COW. There is no layer of "indirection" above this, so that's where a lot of the trickiness comes from, I think. If you have indirect "logical pointers" instead, you can defer a lot of work, things like reclamation, fragmentation, splitting and merging, etc. There are some other interesting design points in the notes[1] I was looking over to refresh myself, but I don't know how some of them relate to ZFS specifically. For example, another thing bcachefs is pretty good at is low tail latency, and the note on eschewing heavyweight transactions/locks is really good example of that. If you want to run 'touch file', you don't need to take a transaction lock, create an inode, then create the dirent, and release the lock. You can instead simply create the inode and then create dirent assuming that they have a proper write barrier between them, and deleting a file is the inverse operation. Then the system can never be left in an inconsistent state, it can only result in a few dangling objects in the key-value store, which can be garbage collected later. The low tail latency is also assisted by having really small amounts of metadata for each inode, just barely a 64-byte cache line IIRC; so some really weird pathological cases like `ls` in a directory with 500k dirents is way faster in bcachefs than elsewhere. I also wonder if that also falls out from some of its basic design; it basically looks like a POSIX filesystem layered on top of a fast, ordered b-tree, so there's the storage layer and the actual filesystem semantics on top. The filesystem layer probably doesn't actually need to record that much in-band metadata for a given inode when a lot of the underlying consistency and performance issues are handled elsewhere. But that's just me thinking out loud. Overall, there are lots of differences in the details even if some of the high-level stuff comes across as the same. [1] https://bcachefs.org/Architecture/ https://bcachefs.org/Architecture/
magicalhippo 3y ago
This is similar to how ZFS does it, and probably others, so hardly unique. In ZFS they call it "records" or "blocks" rather than "extents", and the default value record size is 128KiB.

by tyingq 3y ago

cwillu 3y ago

That's not unique to bcachefs; it's how transparent compression is implemented in every filesystem that implements it that I'm aware of (for example, I just checked NTFS, BTRFS and ZFS).

mastax 3y ago
ZFS compresses records (blocks) independently.
- someplaceguy 3y ago
  A ZFS block is 128 KB by default (smaller if the file is < 128 KB, but can also be larger if you manually configure it). So a ZFS block is even larger than a bcachefs "extent".
  - mastax 3y ago
    Well yes, my point is that (counter to GP) ZFS doesn't combine together multiple blocks into a compression extent, rather it uses larger blocks by default.
    - someplaceguy 3y ago
      ZFS's larger blocks you are referring to are the logical block size, which is the size unit for compression among other things. I don't know much about bcachefs, but as I understand it, a logical block in ZFS is equivalent to a bcachefs extent (in terms of compression, at least) and a ZFS physical block is equivalent to a (possibly empty) set of bcachefs blocks. As an example, in terms of actual on-disk storage, a 128 KB logical block in ZFS can be compressed down to (for example) a 3.5 KB "physical" block and it will only actually consume 3.5 KB of disk space (surprisingly, the block size of a ZFS physical block doesn't have to be a power of 2!). That said, if you have 4 KB-sector disks then you should probably use a physical block size granularity of 4 KB for your ZFS pool rather than 512 bytes. So, regardless of the file size, an on-disk ZFS block can actually occupy anywhere from 0 bytes up to 1 MB, with a granularity of up to 512 bytes, depending on how much it is compressed, what the physical block size granularity of the ZFS pool is and what you've selected as the recordsize property (i.e. the logical block size) for the corresponding filesystem in the pool.
      - rincebrain 3y ago
        16M, not 1M.
        someplaceguy 3y ago
        Thanks for the correction. The `zfsprops` man page should be updated to reflect that...
        rincebrain 3y ago
        It has been. The default was to limit it to 1M in case there were weird edge cases with the larger sizes (16M is the largest it'll encode on disk without a format change, IIRC), with larger sizes behind a tunable, but after a few years of people who knew this twiddling it, the default in 2.2 will be that the tunable is set to 16M.
      - 5e92cb50239222b 3y ago
        > if you have 4 KB-sector disks then you should probably use a physical block size granularity of 4 KB for your ZFS pool rather than 512 bytes According to Allan Jude (OpenZFS & FreeBSD developer), you should never use 512-byte recordsize even on an HDD with 512-byte sectors. Always start from 4 KiBs, probably even from 8. https://klarasystems.com/articles/tuning-recordsize-in-openzfs https://klarasystems.com/articles/tuning-recordsize-in-openz...
deleted 3y ago
[deleted]
aseipp 3y ago
ZFS is block based, not extent based, but blocks can be variable size from my understanding. Btrfs does compress individual blocks at 4KiB granularity, but can use the same zstd dictionary for up-to 128KiB runs of contiguous blocks. I don't know about modern NTFS but IIRC it can't compress logical blocks larger than 4KiB anyway, or couldn't a long time ago, so at that point it's kind of moot when compared to the greater ratios of the competitors. In practice being a true extent-based filesystem with compression and snapshotting is one of bcachefs's distinguishing features (snapshotting is rather obvious in a block-based system by comparison). Other systems work similarly at a high level but the details are of course quite different in practice.
- someplaceguy 3y ago
  > In practice being a true extent-based filesystem with compression and snapshotting is one of bcachefs's distinguishing features (snapshotting is rather obvious in a block-based system by comparison). Other systems work similarly at a high level but the details are of course quite different in practice. How do bcachefs' extents differ from ZFS's logical blocks (whose size are controlled by the dataset's recordsize property) and how do bcachefs' blocks differ from ZFS's physical blocks (whose size granularity are controlled by the ZFS pool's ashift property)?
  - aseipp 3y ago
    My understanding (Kent is in this thread so maybe I'm totally wrong and he can improve on this) is that extents work like any other filesystem but taken to the extreme: they record a variable, contiguous range of bytes, and that's really all there is in bcachefs, there is no other fixed size unit of data for file data to fit into (I suppose beyond whatever is enforced by the underlying drive geometry and btree index); while a ZFS logical block has several underlying physical blocks of smaller size, but ultimately the logical block is the "unit of operation." It's kind of like the difference between your data type being single scalar number and an interval, I think. bcachefs only has extents and all data is recorded as a variable-sized extent. In contrast, ZFS logical blocks have a fixed size, but may be handled "sparsely" by an underlying number of (smaller) physical blocks. So a 128KiB logical block doesn't mean every 4KiB file takes up 128KiB, naturally. But to change 'recordsize' in ZFS, you also need to rewrite the file, since the logical block size won't be retroactively changed. For good reason, of course. But the indirection between physical/logical makes a big difference in the underlying mechanics. Snapshots make a lot of sense in a system that uses logical, fixed-sized blocks as the atomic "unit" of data even if there is an underlying physical block layer. For example, COW is trivial -- page is read, written and marked dirty, then written out to a new copy of the block it came from. The original logical block can be left alone, and the cost is understood as proportional to the underlying number of physical blocks rewritten (and some change to update metadata). But how does COW or snapshotting work in an extent system? In a non-COW system, you can update the data inside the extent in-place since you don't copy, but in a COW system you need to copy or potentially split the extent which could be costly. You may actually want to group extents together if they're for multiple versions of a file when doing COW. There is no layer of "indirection" above this, so that's where a lot of the trickiness comes from, I think. If you have indirect "logical pointers" instead, you can defer a lot of work, things like reclamation, fragmentation, splitting and merging, etc. There are some other interesting design points in the notes[1] I was looking over to refresh myself, but I don't know how some of them relate to ZFS specifically. For example, another thing bcachefs is pretty good at is low tail latency, and the note on eschewing heavyweight transactions/locks is really good example of that. If you want to run 'touch file', you don't need to take a transaction lock, create an inode, then create the dirent, and release the lock. You can instead simply create the inode and then create dirent assuming that they have a proper write barrier between them, and deleting a file is the inverse operation. Then the system can never be left in an inconsistent state, it can only result in a few dangling objects in the key-value store, which can be garbage collected later. The low tail latency is also assisted by having really small amounts of metadata for each inode, just barely a 64-byte cache line IIRC; so some really weird pathological cases like `ls` in a directory with 500k dirents is way faster in bcachefs than elsewhere. I also wonder if that also falls out from some of its basic design; it basically looks like a POSIX filesystem layered on top of a fast, ordered b-tree, so there's the storage layer and the actual filesystem semantics on top. The filesystem layer probably doesn't actually need to record that much in-band metadata for a given inode when a lot of the underlying consistency and performance issues are handled elsewhere. But that's just me thinking out loud. Overall, there are lots of differences in the details even if some of the high-level stuff comes across as the same. [1] https://bcachefs.org/Architecture/ https://bcachefs.org/Architecture/

magicalhippo 3y ago

This is similar to how ZFS does it, and probably others, so hardly unique. In ZFS they call it "records" or "blocks" rather than "extents", and the default value record size is 128KiB.