4 ms·

What I really would like: ext4 with snapshots and transparent compression where it may improve performance.

by marcodiego 3y ago

arjvik 3y ago
so essentially btrfs/bcachefs without the focus on being used for RAID?
- j16sdiz 3y ago
  btrfs don't allow nodatacow with snapshots . I know how these two features have some conflicts, but it meant we can't have database workload with decent performance on btrfs. -- brtfs snapshot just can't scale, the cow have too high performance hit. ZFS handles snapshots with database workload just fine.
  - viraptor 3y ago
    Have you got any recent snapshot benchmark available? I can find only one ancient one. > btrfs don't allow nodatacow with snapshots What's the use case for a snapshot at that point? Isn't it the same as making a copy of the files? I'm not even sure what would this look like... CoW is what enables snapshots. Without CoW you can't get a consistent copy anymore.
    - lmz 3y ago
      I guess it would look like LVM snapshots? CoW only when a snapshot is present.
      - yencabulator 3y ago
        That's how btrfs chattr +C files behave already.
    - derefr 3y ago
      > What's the use case for a snapshot at that point? Isn't it the same as making a copy of the files? As you say, it's a consistent copy. cp(1) won't give you that.
      - viraptor 3y ago
        So I guess you'd have to temporarily turn on CoW, take a snapshot, make sure all the files are fully duplicated, turn off CoW to achieve that. Since you can't flip the setting at runtime right now, that feature seems quite far away.
        j16sdiz 3y ago
        In brtfs, you can't. You need to copy the file over to enable/disable cow
      - tremon 3y ago
        Neither does a filesystem snapshot unless the snapshot is aware of every application's consistency semantics. You can't simply assume that every flush() or other write barrier means that the application data is now in a consistent state on-disk.
        derefr 3y ago
        These days, with production systems being componentized across VMs, abstractions like LVM in use, etc., the filesystem is no longer a fixed "feature of the deploy environment" chosen separately from the needs of the application, that needs to cope with any random thing the production application might do; but rather, a prod deployment is designed as a whole, where you choose the filesystem and plan the use of its features relative to what application you'll be deploying on it. (And, in fact, for a stateful application like a DBMS, you'll probably have one or more separate volumes just for the application state — so the filesystem you use to solve application-state problems doesn't even need to be good at being a rootfs for an OS; it only needs to be good for your application use-case.) Under this paradigm, rather than trying to make a filesystem that understands applications well enough to snapshot them, you instead make applications that have filesystem snapshots as part of their conceptual model. In the lower-effort version of this approach, you have software like Postgres, where the application layer can be told "I'm going to use filesystem tooling to take a consistent snapshot, so make your on-disk state consistent for a while." In PG, you'd call pg_start_backup(), which will flush all pending writes to the table files to disk, and then spool all future writes purely in the WAL journal until the backup completes (i.e. until you tell it pg_stop_backup()) — at which point all those pending changes get replayed out to the table files. In the higher-effort version of this approach, you have software that has one or more CoW filesystems it natively understands and integrates with — where you never directly address the filesystem at all, but rather, you tell the application to take an application level snapshot; and then the application uses the filesystem CoW features, together with its own consistency primitives, to efficiently achieve that (which might not necessarily result in something that's a "filesystem snapshot" from the FS's perspective, but rather just a bunch of individual CoW-cloned files in a directory.) I believe that Oracle DBMS does this, though I might be wrong.
  - toast0 3y ago
    ZFS is cow, so if it works for your load, cow doesn't seem to be the problem? If you're willing to switch OSes, FreeBSD UFS is a more traditional filesystem, with optional snapshots (modifications to files in a snapshot have to be cow, of course)
    - j16sdiz 3y ago
      > ZFS is cow, so if it works for your load, cow doesn't seem to be the problem? Btrfs' implementation of cow, of course. They have no intention to make it work for database workload. When you ask why it's slow, they just ask you to disable cow ( which requires recreating the file -- something you would die to avoid with multi TB database)
  - yencabulator 3y ago
    > btrfs don't allow nodatacow with snapshots . Uhh. "You can't do overwrite-in-place if you want to keep a snapshot copy of the old data" simply makes sense, and you'll find every overwrite-in-place design that supports snapshots will take a write performance hit around the time the snapshot; either the snapshot has to atomically copy the whole data, or the first write after a snapshot can't be overwrite-in-place (or it has to make a separate copy of the original data for the snapshot; similar but worse).
    - Dylan16807 3y ago
      Proper CoW means that every change to a block has to cause copies, not just the first write after a snapshot. That makes a big difference. And "move the snapshotted data out of the way upon writing" is going to give you better performance in a lot of cases.
      - yencabulator 3y ago
        I was replying to this part: > btrfs don't allow nodatacow with snapshots. You can absolutely mix chattr +C with snapshots. It's CoW-when-needed in the face of snapshots, just like everything has to be in order to support snapshots.
        Dylan16807 3y ago
        So they were just wrong in saying it's not supported? You didn't make that clear. And yes I can find many sources saying it's supported.
wmf 3y ago
It's kludgey to graft snapshots and compression onto a classic in-place filesystem, so you'd probably end up with something slower and more complex than ZFS/btrfs/bcachefs. I remember tux2/tux3 was touting its simplicity but I don't know what happened.
- 5e92cb50239222b 3y ago
  Dave Chinner somehow managed to put reflinks on top of XFS, although it's not nearly the same as full filesystem snapshots. I believe there was discussion about implementing proper snapshots, but XFS is very conservatively developed, so that may be many years in the future, if it ever happens.
- throw0101a 3y ago
  > It's kludgey to graft snapshots […] McKusick did this for FreeBSD UFS/FFS: * https://wiki.freebsd.org/ExampleUfsSnapshots https://wiki.freebsd.org/ExampleUfsSnapshots * https://people.freebsd.org/~rse/snapshot/ https://people.freebsd.org/~rse/snapshot/ * https://man.freebsd.org/cgi/man.cgi?query=mksnap_ffs https://man.freebsd.org/cgi/man.cgi?query=mksnap_ffs Various papers at: * http://www.mckusick.com/softdep/ http://www.mckusick.com/softdep/
the8472 3y ago
I think LVM or DM provide snapshot and compression capabilities which you could layer under ext4. I like ext4 for being simple and fast and having options to turn off the unnecessary stuff. It's great for gigabytes of caches, tempfiles, build artifacts, scratch files etc. The bookkeeping and indirection needed for those features would be a waste of CPU cycles for shortlived data.
Nux 3y ago
Check VDO device mapper module.
matheusmoreira 3y ago
I want ext4 to remain rock solid and reliable, and LVM to somehow gain btrfs's flexible block allocator so we can have easily expandable heterogeneous pools of drives. I think by this point it's clear btrfs is never gonna turn into Linux's ZFS so perhaps it'd be wise to implement its good ideas in other systems instead.

What I really would like: ext4 with snapshots and transparent compression where it may improve performance.

arjvik 3y ago

so essentially btrfs/bcachefs without the focus on being used for RAID?

j16sdiz 3y ago
btrfs don't allow nodatacow with snapshots . I know how these two features have some conflicts, but it meant we can't have database workload with decent performance on btrfs. -- brtfs snapshot just can't scale, the cow have too high performance hit. ZFS handles snapshots with database workload just fine.
- viraptor 3y ago
  Have you got any recent snapshot benchmark available? I can find only one ancient one. > btrfs don't allow nodatacow with snapshots What's the use case for a snapshot at that point? Isn't it the same as making a copy of the files? I'm not even sure what would this look like... CoW is what enables snapshots. Without CoW you can't get a consistent copy anymore.
  - lmz 3y ago
    I guess it would look like LVM snapshots? CoW only when a snapshot is present.
    - yencabulator 3y ago
      That's how btrfs chattr +C files behave already.
  - derefr 3y ago
    > What's the use case for a snapshot at that point? Isn't it the same as making a copy of the files? As you say, it's a consistent copy. cp(1) won't give you that.
    - viraptor 3y ago
      So I guess you'd have to temporarily turn on CoW, take a snapshot, make sure all the files are fully duplicated, turn off CoW to achieve that. Since you can't flip the setting at runtime right now, that feature seems quite far away.
      - j16sdiz 3y ago
        In brtfs, you can't. You need to copy the file over to enable/disable cow
    - tremon 3y ago
      Neither does a filesystem snapshot unless the snapshot is aware of every application's consistency semantics. You can't simply assume that every flush() or other write barrier means that the application data is now in a consistent state on-disk.
      - derefr 3y ago
        These days, with production systems being componentized across VMs, abstractions like LVM in use, etc., the filesystem is no longer a fixed "feature of the deploy environment" chosen separately from the needs of the application, that needs to cope with any random thing the production application might do; but rather, a prod deployment is designed as a whole, where you choose the filesystem and plan the use of its features relative to what application you'll be deploying on it. (And, in fact, for a stateful application like a DBMS, you'll probably have one or more separate volumes just for the application state — so the filesystem you use to solve application-state problems doesn't even need to be good at being a rootfs for an OS; it only needs to be good for your application use-case.) Under this paradigm, rather than trying to make a filesystem that understands applications well enough to snapshot them, you instead make applications that have filesystem snapshots as part of their conceptual model. In the lower-effort version of this approach, you have software like Postgres, where the application layer can be told "I'm going to use filesystem tooling to take a consistent snapshot, so make your on-disk state consistent for a while." In PG, you'd call pg_start_backup(), which will flush all pending writes to the table files to disk, and then spool all future writes purely in the WAL journal until the backup completes (i.e. until you tell it pg_stop_backup()) — at which point all those pending changes get replayed out to the table files. In the higher-effort version of this approach, you have software that has one or more CoW filesystems it natively understands and integrates with — where you never directly address the filesystem at all, but rather, you tell the application to take an application level snapshot; and then the application uses the filesystem CoW features, together with its own consistency primitives, to efficiently achieve that (which might not necessarily result in something that's a "filesystem snapshot" from the FS's perspective, but rather just a bunch of individual CoW-cloned files in a directory.) I believe that Oracle DBMS does this, though I might be wrong.
- toast0 3y ago
  ZFS is cow, so if it works for your load, cow doesn't seem to be the problem? If you're willing to switch OSes, FreeBSD UFS is a more traditional filesystem, with optional snapshots (modifications to files in a snapshot have to be cow, of course)
  - j16sdiz 3y ago
    > ZFS is cow, so if it works for your load, cow doesn't seem to be the problem? Btrfs' implementation of cow, of course. They have no intention to make it work for database workload. When you ask why it's slow, they just ask you to disable cow ( which requires recreating the file -- something you would die to avoid with multi TB database)
- yencabulator 3y ago
  > btrfs don't allow nodatacow with snapshots . Uhh. "You can't do overwrite-in-place if you want to keep a snapshot copy of the old data" simply makes sense, and you'll find every overwrite-in-place design that supports snapshots will take a write performance hit around the time the snapshot; either the snapshot has to atomically copy the whole data, or the first write after a snapshot can't be overwrite-in-place (or it has to make a separate copy of the original data for the snapshot; similar but worse).
  - Dylan16807 3y ago
    Proper CoW means that every change to a block has to cause copies, not just the first write after a snapshot. That makes a big difference. And "move the snapshotted data out of the way upon writing" is going to give you better performance in a lot of cases.
    - yencabulator 3y ago
      I was replying to this part: > btrfs don't allow nodatacow with snapshots. You can absolutely mix chattr +C with snapshots. It's CoW-when-needed in the face of snapshots, just like everything has to be in order to support snapshots.
      - Dylan16807 3y ago
        So they were just wrong in saying it's not supported? You didn't make that clear. And yes I can find many sources saying it's supported.

wmf 3y ago

It's kludgey to graft snapshots and compression onto a classic in-place filesystem, so you'd probably end up with something slower and more complex than ZFS/btrfs/bcachefs. I remember tux2/tux3 was touting its simplicity but I don't know what happened.

5e92cb50239222b 3y ago
Dave Chinner somehow managed to put reflinks on top of XFS, although it's not nearly the same as full filesystem snapshots. I believe there was discussion about implementing proper snapshots, but XFS is very conservatively developed, so that may be many years in the future, if it ever happens.
throw0101a 3y ago
> It's kludgey to graft snapshots […] McKusick did this for FreeBSD UFS/FFS: * https://wiki.freebsd.org/ExampleUfsSnapshots https://wiki.freebsd.org/ExampleUfsSnapshots * https://people.freebsd.org/~rse/snapshot/ https://people.freebsd.org/~rse/snapshot/ * https://man.freebsd.org/cgi/man.cgi?query=mksnap_ffs https://man.freebsd.org/cgi/man.cgi?query=mksnap_ffs Various papers at: * http://www.mckusick.com/softdep/ http://www.mckusick.com/softdep/

the8472 3y ago

I think LVM or DM provide snapshot and compression capabilities which you could layer under ext4. I like ext4 for being simple and fast and having options to turn off the unnecessary stuff. It's great for gigabytes of caches, tempfiles, build artifacts, scratch files etc. The bookkeeping and indirection needed for those features would be a waste of CPU cycles for shortlived data.

Nux 3y ago

Check VDO device mapper module.

matheusmoreira 3y ago

I want ext4 to remain rock solid and reliable, and LVM to somehow gain btrfs's flexible block allocator so we can have easily expandable heterogeneous pools of drives. I think by this point it's clear btrfs is never gonna turn into Linux's ZFS so perhaps it'd be wise to implement its good ideas in other systems instead.