4 ms·

Can someone explain to me (a bloody layman) what the advantages/differences are over a traditional file system ext3?

by folli 4y ago

justinlloyd 4y ago
CoW (copy-on-write), deduplication of data blocks, replicas (RAID equivalent), file system snapshots and multiple, tiered caching layers, plus several other features. On a sidenote, I integrated bcachefs as a vmx driver into VMWare ESXi for a CI/CD build server a few years back. The build system ran in VMs and on containers, but the non-essential target directories sat on bcachefs volumes with the caching layer directed first at RAM, then at SSD, then finally at the HDD. Managing all the Unity3D and Unreal caches was amazingly fast across dozens of different SKUs of what the same project. From my project write-up on my LinkedIn: Bcache-like caching layer for VMWare ESXi Reduce latency and read/write delay even if using SSD as your storage. Written in C and inserting itself as a storage tier into VMWare ESXi to handle read & write storage requests this caching system accelerates all accesses to the underlying backing store. Can work in both write-back and write-through caching modes. (Native C kernel device driver)
- nwellinghoff 4y ago
  Cool stuff, do you have a more detailed write up of this? I am interested in learning more!
- pitaj 4y ago
  Does bcachefs support something like raidz2?
attentive 4y ago
checksums, compress, encrypt, snapshots etc. It's a long list. It's better to compare with btrfs
formerly_proven 4y ago
There's a ton of features, but really the most important one is that CoW filesystems tend to extend checksum protection to the actual data you're storing on them. Journaling filesystems generally only protect themselves and not the data.
- FooBarWidget 4y ago
  Is filesystem-level checksums better than dm-integrity which works at the block level?
  - greenicon 4y ago
    With dm-integrity you either have a small hole of a few milliseconds (bitmap mode) or write all data twice (journal mode). When integrating everything into a cow file system you can sidestep the issue, as you basically journal the data through cow anyway.
  - viraptor 4y ago
    The only small advantage I can think of is that the filesystem knows which parts of the drive are being used, so it will not try to recover stale/unused blocks. Same goes for writing - dm-integrity needs to initialise the whole drive and update every single write, but the fs can issue some big TRIMs instead. (Or does integrity know how to trim these days? (Apparently it depends on the usage mode https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/dm-integrity.html https://www.kernel.org/doc/html/latest/admin-guide/device-ma...))
Shish2k 4y ago
The thing that I’m most looking forward to: the ability to have multiple drives of different sizes, and expand the array with new oddly-sized drives (great for a homelab NAS built of spare parts; btrfs can do this, but zfs can't), combined with being able to set some drives as caches (eg having the filesystem automatically store frequently-read data on an SSD and rarely-read data on HDD; zfs can do this, btrfs can't)
mekster 4y ago
I don't know about bcachefs' capability on this but with zfs, you can do an instant snapshot of MySQL/PostgreSQL data directory and call it a backup instead of fighting the tough fight that is database backup with their dump utilities taking a good amount of space and time and not easy to do incremental backups. https://lackofimagination.org/2022/04/our-experience-with-postgresql-on-zfs/ https://lackofimagination.org/2022/04/our-experience-with-po... https://www.percona.com/blog/2017/12/07/hands-look-zfs-with-mysql/ https://www.percona.com/blog/2017/12/07/hands-look-zfs-with-...
pavon 4y ago
I really want a filesystem with snapshots as the foundation of a better backup system. Bcachefs has other useful features like CoW and disk pooling (in fact snapshots weren't even on the table until recently), but in my mind snapshots are a must-have feature these days, like journaling was 20 years ago. Unfortunately, this has been just around the corner on Linux for over a decade now, and the two filesystems that promised to deliver it are unlikely to reach mainstream support on Linux. ZFS has licensing issues, and uses too much memory for a desktop system. BTRFS tried to do too many things, and has had too many reliability issues for me to trust it.
- mekster 4y ago
  Licensing issue is only for distribution. You're free to use it without breaching the license. Not sure why people won't just use it. How is zfs using too much memory? zfs can run on a 2GB server (with some swap). Any laptop would have enough memory to run it. You might want to change "zfs_arc_max" as zfs tries to use half the memory available on a system if you don't set it and don't use deduplication.

Can someone explain to me (a bloody layman) what the advantages/differences are over a traditional file system ext3?

by folli 4y ago

Can someone explain to me (a bloody layman) what the advantages/differences are over a traditional file system ext3?

CoW (copy-on-write), deduplication of data blocks, replicas (RAID equivalent), file system snapshots and multiple, tiered caching layers, plus several other features. On a sidenote, I integrated bcachefs as a vmx driver into VMWare ESXi for a CI/CD build server a few years back. The build system ran in VMs and on containers, but the non-essential target directories sat on bcachefs volumes with the caching layer directed first at RAM, then at SSD, then finally at the HDD. Managing all the Unity3D and Unreal caches was amazingly fast across dozens of different SKUs of what the same project. From my project write-up on my LinkedIn: Bcache-like caching layer for VMWare ESXi Reduce latency and read/write delay even if using SSD as your storage. Written in C and inserting itself as a storage tier into VMWare ESXi to handle read & write storage requests this caching system accelerates all accesses to the underlying backing store. Can work in both write-back and write-through caching modes. (Native C kernel device driver)

nwellinghoff 4y ago
Cool stuff, do you have a more detailed write up of this? I am interested in learning more!
pitaj 4y ago
Does bcachefs support something like raidz2?

attentive 4y ago

checksums, compress, encrypt, snapshots etc. It's a long list. It's better to compare with btrfs

formerly_proven 4y ago

There's a ton of features, but really the most important one is that CoW filesystems tend to extend checksum protection to the actual data you're storing on them. Journaling filesystems generally only protect themselves and not the data.

FooBarWidget 4y ago
Is filesystem-level checksums better than dm-integrity which works at the block level?
- greenicon 4y ago
  With dm-integrity you either have a small hole of a few milliseconds (bitmap mode) or write all data twice (journal mode). When integrating everything into a cow file system you can sidestep the issue, as you basically journal the data through cow anyway.
- viraptor 4y ago
  The only small advantage I can think of is that the filesystem knows which parts of the drive are being used, so it will not try to recover stale/unused blocks. Same goes for writing - dm-integrity needs to initialise the whole drive and update every single write, but the fs can issue some big TRIMs instead. (Or does integrity know how to trim these days? (Apparently it depends on the usage mode https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/dm-integrity.html https://www.kernel.org/doc/html/latest/admin-guide/device-ma...))

Shish2k 4y ago

The thing that I’m most looking forward to: the ability to have multiple drives of different sizes, and expand the array with new oddly-sized drives (great for a homelab NAS built of spare parts; btrfs can do this, but zfs can't), combined with being able to set some drives as caches (eg having the filesystem automatically store frequently-read data on an SSD and rarely-read data on HDD; zfs can do this, btrfs can't)

mekster 4y ago

I don't know about bcachefs' capability on this but with zfs, you can do an instant snapshot of MySQL/PostgreSQL data directory and call it a backup instead of fighting the tough fight that is database backup with their dump utilities taking a good amount of space and time and not easy to do incremental backups. https://lackofimagination.org/2022/04/our-experience-with-postgresql-on-zfs/ https://lackofimagination.org/2022/04/our-experience-with-po... https://www.percona.com/blog/2017/12/07/hands-look-zfs-with-mysql/ https://www.percona.com/blog/2017/12/07/hands-look-zfs-with-...

pavon 4y ago

I really want a filesystem with snapshots as the foundation of a better backup system. Bcachefs has other useful features like CoW and disk pooling (in fact snapshots weren't even on the table until recently), but in my mind snapshots are a must-have feature these days, like journaling was 20 years ago. Unfortunately, this has been just around the corner on Linux for over a decade now, and the two filesystems that promised to deliver it are unlikely to reach mainstream support on Linux. ZFS has licensing issues, and uses too much memory for a desktop system. BTRFS tried to do too many things, and has had too many reliability issues for me to trust it.

mekster 4y ago
Licensing issue is only for distribution. You're free to use it without breaching the license. Not sure why people won't just use it. How is zfs using too much memory? zfs can run on a 2GB server (with some swap). Any laptop would have enough memory to run it. You might want to change "zfs_arc_max" as zfs tries to use half the memory available on a system if you don't set it and don't use deduplication.