3 ms·

so essentially btrfs/bcachefs without the focus on being used for RAID?

• j16sdiz 3y ago
  btrfs don't allow nodatacow with snapshots . I know how these two features have some conflicts, but it meant we can't have database workload with decent performance on btrfs. -- brtfs snapshot just can't scale, the cow have too high performance hit. ZFS handles snapshots with database workload just fine.
  • viraptor 3y ago
    Have you got any recent snapshot benchmark available? I can find only one ancient one. > btrfs don't allow nodatacow with snapshots What's the use case for a snapshot at that point? Isn't it the same as making a copy of the files? I'm not even sure what would this look like... CoW is what enables snapshots. Without CoW you can't get a consistent copy anymore.
    • lmz 3y ago
      I guess it would look like LVM snapshots? CoW only when a snapshot is present.
      • yencabulator 3y ago
        That's how btrfs chattr +C files behave already.
    • derefr 3y ago
      > What's the use case for a snapshot at that point? Isn't it the same as making a copy of the files? As you say, it's a consistent copy. cp(1) won't give you that.
      • viraptor 3y ago
        So I guess you'd have to temporarily turn on CoW, take a snapshot, make sure all the files are fully duplicated, turn off CoW to achieve that. Since you can't flip the setting at runtime right now, that feature seems quite far away.
        • j16sdiz 3y ago
          In brtfs, you can't. You need to copy the file over to enable/disable cow
      • tremon 3y ago
        Neither does a filesystem snapshot unless the snapshot is aware of every application's consistency semantics. You can't simply assume that every flush() or other write barrier means that the application data is now in a consistent state on-disk.
        • derefr 3y ago
          These days, with production systems being componentized across VMs, abstractions like LVM in use, etc., the filesystem is no longer a fixed "feature of the deploy environment" chosen separately from the needs of the application, that needs to cope with any random thing the production application might do; but rather, a prod deployment is designed as a whole, where you choose the filesystem and plan the use of its features relative to what application you'll be deploying on it. (And, in fact, for a stateful application like a DBMS, you'll probably have one or more separate volumes just for the application state — so the filesystem you use to solve application-state problems doesn't even need to be good at being a rootfs for an OS; it only needs to be good for your application use-case.) Under this paradigm, rather than trying to make a filesystem that understands applications well enough to snapshot them, you instead make applications that have filesystem snapshots as part of their conceptual model. In the lower-effort version of this approach, you have software like Postgres, where the application layer can be told "I'm going to use filesystem tooling to take a consistent snapshot, so make your on-disk state consistent for a while." In PG, you'd call pg_start_backup(), which will flush all pending writes to the table files to disk, and then spool all future writes purely in the WAL journal until the backup completes (i.e. until you tell it pg_stop_backup()) — at which point all those pending changes get replayed out to the table files. In the higher-effort version of this approach, you have software that has one or more CoW filesystems it natively understands and integrates with — where you never directly address the filesystem at all, but rather, you tell the application to take an application level snapshot; and then the application uses the filesystem CoW features, together with its own consistency primitives, to efficiently achieve that (which might not necessarily result in something that's a "filesystem snapshot" from the FS's perspective, but rather just a bunch of individual CoW-cloned files in a directory.) I believe that Oracle DBMS does this, though I might be wrong.
        • Dylan16807 3y ago
  • toast0 3y ago
    ZFS is cow, so if it works for your load, cow doesn't seem to be the problem? If you're willing to switch OSes, FreeBSD UFS is a more traditional filesystem, with optional snapshots (modifications to files in a snapshot have to be cow, of course)
    • j16sdiz 3y ago
      > ZFS is cow, so if it works for your load, cow doesn't seem to be the problem? Btrfs' implementation of cow, of course. They have no intention to make it work for database workload. When you ask why it's slow, they just ask you to disable cow ( which requires recreating the file -- something you would die to avoid with multi TB database)
  • yencabulator 3y ago
    > btrfs don't allow nodatacow with snapshots . Uhh. "You can't do overwrite-in-place if you want to keep a snapshot copy of the old data" simply makes sense, and you'll find every overwrite-in-place design that supports snapshots will take a write performance hit around the time the snapshot; either the snapshot has to atomically copy the whole data, or the first write after a snapshot can't be overwrite-in-place (or it has to make a separate copy of the original data for the snapshot; similar but worse).
    • Dylan16807 3y ago
      Proper CoW means that every change to a block has to cause copies, not just the first write after a snapshot. That makes a big difference. And "move the snapshotted data out of the way upon writing" is going to give you better performance in a lot of cases.
      • yencabulator 3y ago
        I was replying to this part: > btrfs don't allow nodatacow with snapshots. You can absolutely mix chattr +C with snapshots. It's CoW-when-needed in the face of snapshots, just like everything has to be in order to support snapshots.
        • Dylan16807 3y ago
          So they were just wrong in saying it's not supported? You didn't make that clear. And yes I can find many sources saying it's supported.