11 ms·

ZFS 2.2.0 (RC): Block Cloning merged

286 points by turrini 3y ago

miohtama 3y ago
What are applications that benefit from block cloning?
- thrill 3y ago
  FTFA: "Block Cloning allows to clone a file (or a subset of its blocks) into another (or the same) file by just creating additional references to the data blocks without copying the data itself. Block Cloning can be described as a fast, manual deduplication."
- mustache_kimono 3y ago
  Excited, because in addition to ref copies/clones, httm will use this feature, if available (I've already done some work to implement), for its `--roll-forward` operation, and for faster file recoveries from snapshots [0]. As I understand it, there will be no need to copy any data from the same dataset, and this includes all snapshots. Blocks written to the live dataset can just be references to the underlying blocks, and no additional space will need be used. Imagine being able to continuously switch a file or a dataset back to a previous state extremely quickly without a heavy weight clone, or a rollback, etc. Right now, httm simply diff copies the blocks for file recovery and roll-forward. For further details, see the man page entry for `--roll-forward`, and the link to the httm GitHub below: --roll-forward="snap_name" traditionally 'zfs rollback' is a destructive operation, whereas httm roll-forward is non-destructive. httm will copy only the blocks and file metadata that have changed since a specified snapshot, from that snapshot, to its live dataset. httm will also take two precautionary snapshots, one before and one after the copy. Should the roll forward fail for any reason, httm will roll back to the pre-execution state. Note: This is a ZFS only option which requires super user privileges. [0]: https://github.com/kimono-koans/httm https://github.com/kimono-koans/httm
- vovin 3y ago
  This is huge. One practical application is fast recovery of a file from past snapshot without using any additional space. I use ZFS dataset for my vCenter datastore (storing my vmdk files). In case of need to launch a clone from a past state one could use a block cloning to bring past vmdk file without the need to actually copy the file - it saves both space and time to make such clone.
  - bithavoc 3y ago
    Can you elaborate a bit more on how you use ZFS with vCenter? How do you mount it?
    - redundantly 3y ago
      They're likely using a NAS (e.g., FreeNAS/TrueNAS) that uses ZFS for the underlying storage then sharing that storage by either NFS or iSCSI with their vSphere cluster. In some rare cases FC is being used instead of NFS or iSCSI.
- ikiris 3y ago
  I think the big piece is native overlayfs so k8 setups get a bit simpler.
- philsnow 3y ago
  It seems kind of like hard linking but with copy-on-write for the underlying data, so you'll get near-instant file copies and writing into the middle of them will also be near-instant. All of this happens under the covers already if you have dedup turned on, but this allows utilities (gnu cp might be taught to opportunistically and transparently use the new clone zfs syscalls, because there is no downside and only upside) and applications to tell zfs that "these blocks are going to be the same as those" without zfs needing to hash all the new blocks and compare them. Aditionally, for finer control, ranges of blocks can be cloned, not just entire files. I can't tell from the github issue, can this manual dedup / block cloning be turned on if you're not already using dedup on a dataset? Last time I set up zfs, I was warned that dedup took gobs of memory, so I didn't turn it on.
  - nabla9 3y ago
    Gnu cp --reflink. >When --reflink[=always] is specified, perform a lightweight copy, where the data blocks are copied only when modified. If this is not possible the copy fails, or if --reflink=auto is specified, fall back to a standard copy. Use --reflink=never to ensure a standard copy is performed."
  - rincebrain 3y ago
    It's orthogonal to dedup being on or off, and as someone else said, it's more or less the same underlying semantics you would expect from cp --reflink anywhere. Also, as mentioned, on Linux, it's not wired up with any interface to be used at all right now.
- aardvark179 3y ago
  It can be a really convenient way to snapshot something if you can arrange some point at which everything is synced to disk. Get to that point, make your new files that start sharing all their blocks, and then let your main db process (or whatever) continue on as normal.
- danudey 3y ago
  As others have said: block cloning (the underlying technology that enables copy-on-write) allows you to 'copy' a file without reading all of the data and re-writing it. For example, if you have a 1 GB file and you want to make a copy of it, you need to read the whole file (all at once or in parts) and then write the whole new file (all at once or in parts). This results in 1 GB of reads and 1 GB of writes. Obviously the slower (or more overloaded) your storage media is, the longer this takes. With block cloning, you simply tell the OS "I want this file A to be a copy of this file B" and it creates a new "file" that references all the blocks in the old "file". Given that a "file" on a filesystem is just a list of blocks that make up the data in that file, you can create a new "file" which has pointers to the same blocks as the old "file". This is a simple system call (or a few system calls), and as such isn't much more intensive than simply renaming a file instead of copying it. At my previous job we did builds for our software. This required building the BIOS, kernel, userspace, generating the UI, and so on. These builds required pulling down 10+ GB of git repositories (the git data itself, the checkout, the LFS binary files, external vendor SDKs), and then a large amount of build artifacts on top of that. We also needed to do this build for 80-100 different product models, for both release and debug versions. This meant 200+ copies of the source code alone (not to mention build artifacts and intermediate products), and because of disk space limitations this meant we had to dramatically reduce the number of concurrent builds we could run. The solution we came up with was something like: 1. Check out the source code 2. Create an overlayfs filesystem to mount into each build space 3. Do the build 4. Tear down the overlayfs filesystem This was problematic if we weren't able to mount the filesystem, if we weren't able to unmount the filesystem (because of hanging file descriptors or processes), and so on. Lots of moving parts, lots of `sudo` commands in the scripts, and so on. Copy-on-write would have solved this for us by accomplishing the same thing; we could simply do the following: 1. Check out the source code 2. Have each build process simply `cp -R --reflink=always source/ build_root/`; this would be instantaneous and use no new disk space. 3. Do the build 4. `rm -rf build_root` Fewer moving parts, no root access required, generally simpler all around.
- the8472 3y ago
  Any copy command. On-demand deduplication managed by userspace. https://man7.org/linux/man-pages/man2/ioctl_fideduperange.2.html https://man7.org/linux/man-pages/man2/ioctl_fideduperange.2.... https://man7.org/linux/man-pages/man2/copy_file_range.2.html https://man7.org/linux/man-pages/man2/copy_file_range.2.html https://github.com/markfasheh/duperemove https://github.com/markfasheh/duperemove
vlovich123 3y ago
Do Btrfs or ext4 offer this?
- wtallis 3y ago
  This feature is basically the same as what underpins the reflink feature that btrfs has supported approximately forever and xfs has supported for at least several years.
  - mustache_kimono 3y ago
    Does anyone know whether btrfs or XFS support reflinks from snapshot datasets?
    - Dylan16807 3y ago
      I can confirm BTRFS yes, but note that source and destination need to be on the same mount point before kernel 5.18
    - ComputerGuru 3y ago
      XFS doesn't have native snapshot support, though?
    - danudey 3y ago
      XFS doesn't have snapshot support, so the short answer there is no.
      - mustache_kimono 3y ago
        Shows what I know about XFS. Thanks!
        plq 3y ago
        This is only a tangent given we are talking about snapshots and reflink, but just wanted to mention that LVM has snapshots, so if you need XFS snapshots, create the XFS filesystem on top of an LVM logical volume.
lockhouse 3y ago
Anyone here using ZFS in production these days? If so what OS and implementation? What have been your experiences or gotchas you experienced?
- noaheverett 3y ago
  Running in production for about 3 years with Ubuntu 20.04 / zfs 0.8.3. ZFS is being used as the datastore for a cluster of LXD/LXC instances over multiple physical hosts. I have the OS setup on its own dedicated drive and ZFS striped/cloned over 4 NVMe drives. No gotchas / issues, works well, easy to setup. I am looking forward to the Direct IO speed improvements for NVMe drives with https://github.com/openzfs/zfs/pull/10018 https://github.com/openzfs/zfs/pull/10018 edit: one thing I forgot to mention is, when creating your pool make sure to import your drives by ID (zpool import -d /dev/disk/by-id/ <poolname>) instead of name in case name assignments change somehow [1] [1] https://superuser.com/questions/1732532/zfs-disk-drive-letter-changed-how-to-reimport-by-id https://superuser.com/questions/1732532/zfs-disk-drive-lette...
  - throw0101c 3y ago
    > I am looking forward to the Direct IO speed improvements for NVMe drives with https://github.com/openzfs/zfs/pull/10018 https://github.com/openzfs/zfs/pull/10018 See also "Scaling ZFS for NVMe" by Allan Jude at EuroBSDcon 2022: * https://www.youtube.com/watch?v=v8sl8gj9UnA https://www.youtube.com/watch?v=v8sl8gj9UnA
    - noaheverett 3y ago
      Sweet, I appreciate the link!
  - doctor_eval 3y ago
    Sorry if this is a newbie question - but what are you using to share the ZFS file system between physical hosts? I’ve been out of this particular game for a long time.
albertzeyer 3y ago
Also see this issue: https://github.com/openzfs/zfs/issues/405 https://github.com/openzfs/zfs/issues/405 > It is in FreeBSD main branch now, but disabled by default just to be safe till after 14.0 released, where it will be included. Can be enabled with loader tunable there. > more code is needed on the ZFS side for Linux integration. A few people are looking at it AFAIK.
GauntletWizard 3y ago
> Note: currently it is not possible to clone blocks between encrypted datasets, even if those datasets use the same encryption key (this includes snapshots of encrypted datasets). Cloning blocks between datasets that use the same keys should be possible and should be implemented in the future. Once this is ready, I am going to subdivide my user homedir much more than it already is. The biggest obstacle in the way of this has been that it would waste a bunch of space until the snapshots were done rolling over, which for me is a long time (I keep weekly snapshots of my homedir for a year).
- yjftsjthsd-h 3y ago
  Is there a benefit to breaking up your home directory?
  - GauntletWizard 3y ago
    Controlling the rate and location of snapshots, mostly. I've broken out some kinds of datasets (video archives) but not others historically (music). It doesn't matter that much, but I want to split some more chunks out.
    - yjftsjthsd-h 3y ago
      Fair enough. I've personally slowly moved to a smaller number of filesystems, but if you're actually handling snapshots differently per-area then it makes sense (indeed, one of the reasons I'm consolidating is the realization that personally I'm almost never going to snapshot/restore things separately).
  - someplaceguy 3y ago
    Not sure why you'd want to do that to your home directory usually, but it depends on what you store in it and how you use it, really. In general, breaking up a filesystem into multiple ones in ZFS is mostly useful for making filesystem management more fine-grained, as a filesystem/dataset in ZFS is the unit of management for most properties and operations (snapshots, clones, compression and checksum algorithms, quotas, encryption, dedup, send/recv, ditto copies, etc) as well as their inheritance and space accounting. In terms of filesystem management, there aren't many downsides to breaking up a filesystem (within reason), as most properties and the most common operations can be shared between all sub-filesystems if they are part of the same inherited tree (which doesn't necessarily have to correspond to the mountpoint tree!). As far as I know, the major downsides by far were that 1) you couldn't quickly move a file from one dataset to another, i.e. `mv` would be forced to do a full copy of the file contents rather than just do a cheap rename, and 2) in terms of disk space, moving a file between filesystems would be equivalent to copying the file and deleting the original, which could be terrible if you use snapshots as it would lead to an additional space consumption of a full new file's worth of disk space. In principle, both of these downsides should be fixed with this new block cloning feature and AFAIU the only tradeoffs would be some amount of increased overhead when freeing data (which should be zero overhead if you don't have many of these cloned blocks being shared anymore), and the low maturity of this code (i.e. higher chance of running into bugs) due to being so new.
Pxtl 3y ago
Filesystem-level de-duplication is scary as hell as a concept, but also sounds amazing, especially doing it at copy-time so you don't have to opt-in to scanning to deduplicate. Is this common in filesystems? Or is ZFS striking out new ground here? I'm not really an under-the-OS-hood kinda guy.
- yjftsjthsd-h 3y ago
  > Filesystem-level de-duplication is scary as hell as a concept What's scary about it? You have to track references, but it doesn't seem that hard compared to everything else going on in ZFS et al. > Is this common in filesystems? Or is ZFS striking out new ground here? At least BTRFS does approximately the same.
  - cesarb 3y ago
    > > Filesystem-level de-duplication is scary as hell as a concept > What's scary about it? It's scary because there's only one copy when you might have expected two. A single bad block could lose both "copies" at once.
    - ForkMeOnTinder 3y ago
      Copying a file isn't great protection against bad blocks. Modern SSDs, when they fail, tend to fail catastrophically (the whole device dies all at once), rather than dying one block at a time. If you care about the data, back it up on a separate piece of hardware.
    - grepfru_it 3y ago
      The file system metadata is redundant and on a correctly configured ZFS system your error correction is isolated and can be redundant as well
    - Filligree 3y ago
      Disks die all the time anyway. If you want to keep your data, you should have at least two-disk redundancy. In which case bad blocks won't kill anything.
uvatbc 3y ago
Technically, yes: through the use of Truenas that gives us API access to iscsi on ZFS.
- deleted 3y ago
  [deleted]
rossmohax 3y ago
Does ZFS or any other FS offer special operations which DB engine like RocksDB, SQLite or PostgreSQL could benefit from if they decided to target that FS specifically?
- magicalhippo 3y ago
  Internally, ZFS is kinda like an object store[1], and there was a project trying to expose the ZFS internals through an object store API rather than through a filesystem API. Sadly I can't seem to find the presentation or recall the name of the project. On the other hand, looking at for example RocksDB[2]: File system operations are not atomic, and are susceptible to inconsistencies in the event of system failure. Even with journaling turned on, file systems do not guarantee consistency on unclean restart. POSIX file system does not support atomic batching of operations either. Hence, it is not possible to rely on metadata embedded in RocksDB datastore files to reconstruct the last consistent state of the RocksDB on restart. RocksDB has a built-in mechanism to overcome these limitations of POSIX file system [...] ZFS does provide atomic operations internally[1], so if exposed it seems something like RocksDB could take advantage of that and forego all the complexity mentioned above. How much that would help I don't know though, but seems potentially interesting at first glance. [1]: https://youtu.be/MsY-BafQgj4?t=442 https://youtu.be/MsY-BafQgj4?t=442 [2]: https://github.com/facebook/rocksdb/wiki/MANIFEST https://github.com/facebook/rocksdb/wiki/MANIFEST
- macdice 3y ago
  https://commitfest.postgresql.org/43/4338/ https://commitfest.postgresql.org/43/4338/
ludde 3y ago
A M A Z I N G Have been looking forward to this for years! This is so much better than automatically doing dedup and the RAM overhead that entails. Doing offline/RAM+in memory dedup size optimizations seem like a really good optimization path. In the spirit of also paying only what you use and not the rest. Edit: What's the RAM overhead of this? Is it ~64B per 128kB deduped block or what's the magnitude of things?
- mlyle 3y ago
  > Edit: What's the RAM overhead of this? Is it ~64B per 128kB deduped block or what's the magnitude of things? No real memory impact. There's a regions table that uses 128k of memory per terabyte of total storage (and may be a bit more in the future). So for your 10 petabyte pool using deduping, you'd better have an extra gigabyte of RAM. But erasing files can potentially be twice as expensive in IOPS, even if not deduped. They try to prevent this.
  - ludde 3y ago
    If I manually deduplicate/copy a 1GB file, how many such offset/refcount tuples would be created in RAM? Just one for the whole file, or one per underlying 128kB block?
    - rincebrain 3y ago
      n.b. I am not pjd, this is from memory and may be wrong. The answer to that is messy, but basically, there's a table that should be kept in memory for "is there anything reflinked at all in this logical range on disk", and that covers large spans, so how many entries would depend on how contiguous your data logically was on disk; the actual precise mapping list per-vdev doesn't need to be kept continuously in memory, just the more coarse table, so that saves you a fair bit on memory requirements.
    - mlyle 3y ago
      They don't go in ram. You have one 64 bit refcount number for every 64MB of storage irrespective of how much is deduplicated; this is a reduction factor of 8M, or 128k per terabyte stored. This largely exists so you can avoid doing extra work on free.
dark-star 3y ago
Wow, I was under the impression that this had long been implemented already (as it's already in btrfs and other commercial file systems) Awesome!
- mgerdts 3y ago
  It has been in the Solaris version of zfs for a long time as well. This came a few years after the Oracle-imposed fork. https://blogs.oracle.com/solaris/post/reflink3c-what-is-it-why-do-i-care-and-how-can-i-use-it https://blogs.oracle.com/solaris/post/reflink3c-what-is-it-w...
  - rincebrain 3y ago
    reflink is implemented on Oracle Solaris; how it works internally I don't think Oracle has ever commented, since implementing post-facto dedup on a filesystem that assumes data once written is unmodified is a bit Spicy(tm) - you can look at the BRT talk from the OpenZFS dev summit or other talks about it at the leadership meetings to get some idea why it's so exciting.
- abwizz 3y ago
  implying that zfs is not a commercial filesystem made me chuckle
XorNot 3y ago
Finally! This is exactly what I want for handling photos and media on my home server. We have phone's dumping their camera rolls onto the machine via Syncthing, but the easiest way to share content safely is just to make copies of the file and go "storage is cheap". But the reality has always been closer to "storage should handle deduplication" - with this feature landing that can finally be a reality. EDIT: Not to mention, this should make ZFS best-in-class for handling container filesystems. Deduplicating all the files in the individual layers means you can pretty much dispense with awkwardly trying to maintain inheritance trees.
- thinkloop 3y ago
  Would you mind expanding on that - what mechanism/app on the phone would share the files?
  - XorNot 3y ago
    Syncthing - the android app can share your camera roll folder. So that syncs to my server, and my desktop/laptop. I drag files around on there when I want them deleted off my phone and archived somewhere. Me and my wife share a syncthing folder between us when we want to send files to each other. All of this produces a fair number of duplicate files, particularly if you have backups turned on in case of deletes. Offline dedupe basically makes all of that free though - duplicate files on the server or in backup dirs are no longer a problem.
- tecleandor 3y ago
  I also was thinking about using this for something similar: organization. I have datasets with unorganized data dumps: photos, music, torrents... Organizing that in shares means duplicating files...
KirillPanov 3y ago
Didn't btrfs have this like a ~decade ago?
- LeoPanthera 3y ago
  Yes, although be careful, because running "defragment" will un-clone and therefore duplicate all your identical blocks.
tecleandor 3y ago
I also see, VERY interesting: - Linux container support (So overlayfs is ZFS aware) - BLAKE3 checksums (So faster checksums) - Zstd early abort (I though Zstd already had early abort! This means you can enable Zstd compression by default, and if the data isn't compressible, it aborts the operation and stores it uncompressed, avoiding the extra CPU use. - Fully adaptive ARC eviction (Separate adaptive ARC for data and for metadata. Until now metadata ARC was almost FIFO and not efficient) - Prefetch improvements (That's always nice)
- someplaceguy 3y ago
  > This means you can enable Zstd compression by default, and if the data isn't compressible, it aborts the operation and stores it uncompressed, avoiding the extra CPU use This was already implemented for all compression algorithms since ZFS was released as open source in 2005, at least (and even though zstd didn't yet exist then, it still benefited from this feature when it was integrated into ZFS). The feature you describe only saves CPU time on decompression, but still requires ZFS to try to compress all data before storing it on disk. This new early abort feature actually seems to be even more clever/sophisticated, because it also saves a lot of CPU time when compressing, not just descompressing. Usually you are forced to select a fast compression level so that your performance doesn't degrade too much for little benefit, because if you had a higher compression level, ZFS would always be paying a high CPU cost for trying to compress data which often is not even compressible (since a lot of data on a typical filesystem belongs to large files, which are often not compressible). However, with early abort, you can select a higher compression level (zstd-3 or higher) without paying a penalty for spending a lot of time trying to compress uncompressible data. The way it seems to work is that ZFS first tries to compress your data with LZ4, which is an extremely fast compression algorithm. If LZ4 succeeds in compressing the data, then ZFS discards the LZ4-compressed data and recompresses it with your chosen higher compression level, knowing that it will be worth it. If LZ4 can't compress the data, then ZFS doesn't even try to compress with the slower algorithm, it just stores the data uncompressed. This almost completely avoids paying the CPU penalty for trying to compress data that cannot be compressed, and only results in about 9% less compression savings than if you had simply chosen the higher compression level without early abort. The actual algorithm is even more clever: in-between the LZ4 heuristic and a zstd-3 or higher compression level, there is an intermediate zstd-1 heuristic that is also done to see whether the data is compressible. This intermediate heuristic seems to almost completely eliminate the 9% compression savings cost while still being much faster than trying to compress all data with zstd-3 or higher, since zstd-1 is also very fast. In the end, you get almost all the benefits of a higher compression level without paying almost any cost for wasting CPU cycles trying to compress uncompressible data!
girishso 3y ago
Pardon my ignorance, but isn’t this same as btrfs snapshots? Any advantage of using zfs over btrfs?
- abwizz 3y ago
  unlikely the same but similar in some aspects. likely zfs and btrfs have some similarities (copy on write) and some differences (integration with volume manager). if you're interested, i'm sure you'll find more.
renewiltord 3y ago
Is the overlayfs/ZFS integration first-class? (i.e. COW on overlayfs is implemented at block layer on ZFS?)

286 points by turrini 3y ago

miohtama 3y ago

What are applications that benefit from block cloning?

thrill 3y ago
FTFA: "Block Cloning allows to clone a file (or a subset of its blocks) into another (or the same) file by just creating additional references to the data blocks without copying the data itself. Block Cloning can be described as a fast, manual deduplication."
mustache_kimono 3y ago
Excited, because in addition to ref copies/clones, httm will use this feature, if available (I've already done some work to implement), for its `--roll-forward` operation, and for faster file recoveries from snapshots [0]. As I understand it, there will be no need to copy any data from the same dataset, and this includes all snapshots. Blocks written to the live dataset can just be references to the underlying blocks, and no additional space will need be used. Imagine being able to continuously switch a file or a dataset back to a previous state extremely quickly without a heavy weight clone, or a rollback, etc. Right now, httm simply diff copies the blocks for file recovery and roll-forward. For further details, see the man page entry for `--roll-forward`, and the link to the httm GitHub below: --roll-forward="snap_name" traditionally 'zfs rollback' is a destructive operation, whereas httm roll-forward is non-destructive. httm will copy only the blocks and file metadata that have changed since a specified snapshot, from that snapshot, to its live dataset. httm will also take two precautionary snapshots, one before and one after the copy. Should the roll forward fail for any reason, httm will roll back to the pre-execution state. Note: This is a ZFS only option which requires super user privileges. [0]: https://github.com/kimono-koans/httm https://github.com/kimono-koans/httm
vovin 3y ago
This is huge. One practical application is fast recovery of a file from past snapshot without using any additional space. I use ZFS dataset for my vCenter datastore (storing my vmdk files). In case of need to launch a clone from a past state one could use a block cloning to bring past vmdk file without the need to actually copy the file - it saves both space and time to make such clone.
- bithavoc 3y ago
  Can you elaborate a bit more on how you use ZFS with vCenter? How do you mount it?
  - redundantly 3y ago
    They're likely using a NAS (e.g., FreeNAS/TrueNAS) that uses ZFS for the underlying storage then sharing that storage by either NFS or iSCSI with their vSphere cluster. In some rare cases FC is being used instead of NFS or iSCSI.
ikiris 3y ago
I think the big piece is native overlayfs so k8 setups get a bit simpler.
philsnow 3y ago
It seems kind of like hard linking but with copy-on-write for the underlying data, so you'll get near-instant file copies and writing into the middle of them will also be near-instant. All of this happens under the covers already if you have dedup turned on, but this allows utilities (gnu cp might be taught to opportunistically and transparently use the new clone zfs syscalls, because there is no downside and only upside) and applications to tell zfs that "these blocks are going to be the same as those" without zfs needing to hash all the new blocks and compare them. Aditionally, for finer control, ranges of blocks can be cloned, not just entire files. I can't tell from the github issue, can this manual dedup / block cloning be turned on if you're not already using dedup on a dataset? Last time I set up zfs, I was warned that dedup took gobs of memory, so I didn't turn it on.
- nabla9 3y ago
  Gnu cp --reflink. >When --reflink[=always] is specified, perform a lightweight copy, where the data blocks are copied only when modified. If this is not possible the copy fails, or if --reflink=auto is specified, fall back to a standard copy. Use --reflink=never to ensure a standard copy is performed."
- rincebrain 3y ago
  It's orthogonal to dedup being on or off, and as someone else said, it's more or less the same underlying semantics you would expect from cp --reflink anywhere. Also, as mentioned, on Linux, it's not wired up with any interface to be used at all right now.
aardvark179 3y ago
It can be a really convenient way to snapshot something if you can arrange some point at which everything is synced to disk. Get to that point, make your new files that start sharing all their blocks, and then let your main db process (or whatever) continue on as normal.
danudey 3y ago
As others have said: block cloning (the underlying technology that enables copy-on-write) allows you to 'copy' a file without reading all of the data and re-writing it. For example, if you have a 1 GB file and you want to make a copy of it, you need to read the whole file (all at once or in parts) and then write the whole new file (all at once or in parts). This results in 1 GB of reads and 1 GB of writes. Obviously the slower (or more overloaded) your storage media is, the longer this takes. With block cloning, you simply tell the OS "I want this file A to be a copy of this file B" and it creates a new "file" that references all the blocks in the old "file". Given that a "file" on a filesystem is just a list of blocks that make up the data in that file, you can create a new "file" which has pointers to the same blocks as the old "file". This is a simple system call (or a few system calls), and as such isn't much more intensive than simply renaming a file instead of copying it. At my previous job we did builds for our software. This required building the BIOS, kernel, userspace, generating the UI, and so on. These builds required pulling down 10+ GB of git repositories (the git data itself, the checkout, the LFS binary files, external vendor SDKs), and then a large amount of build artifacts on top of that. We also needed to do this build for 80-100 different product models, for both release and debug versions. This meant 200+ copies of the source code alone (not to mention build artifacts and intermediate products), and because of disk space limitations this meant we had to dramatically reduce the number of concurrent builds we could run. The solution we came up with was something like: 1. Check out the source code 2. Create an overlayfs filesystem to mount into each build space 3. Do the build 4. Tear down the overlayfs filesystem This was problematic if we weren't able to mount the filesystem, if we weren't able to unmount the filesystem (because of hanging file descriptors or processes), and so on. Lots of moving parts, lots of `sudo` commands in the scripts, and so on. Copy-on-write would have solved this for us by accomplishing the same thing; we could simply do the following: 1. Check out the source code 2. Have each build process simply `cp -R --reflink=always source/ build_root/`; this would be instantaneous and use no new disk space. 3. Do the build 4. `rm -rf build_root` Fewer moving parts, no root access required, generally simpler all around.
the8472 3y ago
Any copy command. On-demand deduplication managed by userspace. https://man7.org/linux/man-pages/man2/ioctl_fideduperange.2.html https://man7.org/linux/man-pages/man2/ioctl_fideduperange.2.... https://man7.org/linux/man-pages/man2/copy_file_range.2.html https://man7.org/linux/man-pages/man2/copy_file_range.2.html https://github.com/markfasheh/duperemove https://github.com/markfasheh/duperemove

vlovich123 3y ago

Do Btrfs or ext4 offer this?

wtallis 3y ago
This feature is basically the same as what underpins the reflink feature that btrfs has supported approximately forever and xfs has supported for at least several years.
- mustache_kimono 3y ago
  Does anyone know whether btrfs or XFS support reflinks from snapshot datasets?
  - Dylan16807 3y ago
    I can confirm BTRFS yes, but note that source and destination need to be on the same mount point before kernel 5.18
  - ComputerGuru 3y ago
    XFS doesn't have native snapshot support, though?
  - danudey 3y ago
    XFS doesn't have snapshot support, so the short answer there is no.
    - mustache_kimono 3y ago
      Shows what I know about XFS. Thanks!
      - plq 3y ago
        This is only a tangent given we are talking about snapshots and reflink, but just wanted to mention that LVM has snapshots, so if you need XFS snapshots, create the XFS filesystem on top of an LVM logical volume.

lockhouse 3y ago

Anyone here using ZFS in production these days? If so what OS and implementation? What have been your experiences or gotchas you experienced?

noaheverett 3y ago
Running in production for about 3 years with Ubuntu 20.04 / zfs 0.8.3. ZFS is being used as the datastore for a cluster of LXD/LXC instances over multiple physical hosts. I have the OS setup on its own dedicated drive and ZFS striped/cloned over 4 NVMe drives. No gotchas / issues, works well, easy to setup. I am looking forward to the Direct IO speed improvements for NVMe drives with https://github.com/openzfs/zfs/pull/10018 https://github.com/openzfs/zfs/pull/10018 edit: one thing I forgot to mention is, when creating your pool make sure to import your drives by ID (zpool import -d /dev/disk/by-id/ <poolname>) instead of name in case name assignments change somehow [1] [1] https://superuser.com/questions/1732532/zfs-disk-drive-letter-changed-how-to-reimport-by-id https://superuser.com/questions/1732532/zfs-disk-drive-lette...
- throw0101c 3y ago
  > I am looking forward to the Direct IO speed improvements for NVMe drives with https://github.com/openzfs/zfs/pull/10018 https://github.com/openzfs/zfs/pull/10018 See also "Scaling ZFS for NVMe" by Allan Jude at EuroBSDcon 2022: * https://www.youtube.com/watch?v=v8sl8gj9UnA https://www.youtube.com/watch?v=v8sl8gj9UnA
  - noaheverett 3y ago
    Sweet, I appreciate the link!
- doctor_eval 3y ago
  Sorry if this is a newbie question - but what are you using to share the ZFS file system between physical hosts? I’ve been out of this particular game for a long time.

albertzeyer 3y ago

Also see this issue: https://github.com/openzfs/zfs/issues/405 https://github.com/openzfs/zfs/issues/405 > It is in FreeBSD main branch now, but disabled by default just to be safe till after 14.0 released, where it will be included. Can be enabled with loader tunable there. > more code is needed on the ZFS side for Linux integration. A few people are looking at it AFAIK.

GauntletWizard 3y ago

> Note: currently it is not possible to clone blocks between encrypted datasets, even if those datasets use the same encryption key (this includes snapshots of encrypted datasets). Cloning blocks between datasets that use the same keys should be possible and should be implemented in the future. Once this is ready, I am going to subdivide my user homedir much more than it already is. The biggest obstacle in the way of this has been that it would waste a bunch of space until the snapshots were done rolling over, which for me is a long time (I keep weekly snapshots of my homedir for a year).

yjftsjthsd-h 3y ago
Is there a benefit to breaking up your home directory?
- GauntletWizard 3y ago
  Controlling the rate and location of snapshots, mostly. I've broken out some kinds of datasets (video archives) but not others historically (music). It doesn't matter that much, but I want to split some more chunks out.
  - yjftsjthsd-h 3y ago
    Fair enough. I've personally slowly moved to a smaller number of filesystems, but if you're actually handling snapshots differently per-area then it makes sense (indeed, one of the reasons I'm consolidating is the realization that personally I'm almost never going to snapshot/restore things separately).
- someplaceguy 3y ago
  Not sure why you'd want to do that to your home directory usually, but it depends on what you store in it and how you use it, really. In general, breaking up a filesystem into multiple ones in ZFS is mostly useful for making filesystem management more fine-grained, as a filesystem/dataset in ZFS is the unit of management for most properties and operations (snapshots, clones, compression and checksum algorithms, quotas, encryption, dedup, send/recv, ditto copies, etc) as well as their inheritance and space accounting. In terms of filesystem management, there aren't many downsides to breaking up a filesystem (within reason), as most properties and the most common operations can be shared between all sub-filesystems if they are part of the same inherited tree (which doesn't necessarily have to correspond to the mountpoint tree!). As far as I know, the major downsides by far were that 1) you couldn't quickly move a file from one dataset to another, i.e. `mv` would be forced to do a full copy of the file contents rather than just do a cheap rename, and 2) in terms of disk space, moving a file between filesystems would be equivalent to copying the file and deleting the original, which could be terrible if you use snapshots as it would lead to an additional space consumption of a full new file's worth of disk space. In principle, both of these downsides should be fixed with this new block cloning feature and AFAIU the only tradeoffs would be some amount of increased overhead when freeing data (which should be zero overhead if you don't have many of these cloned blocks being shared anymore), and the low maturity of this code (i.e. higher chance of running into bugs) due to being so new.

Pxtl 3y ago

Filesystem-level de-duplication is scary as hell as a concept, but also sounds amazing, especially doing it at copy-time so you don't have to opt-in to scanning to deduplicate. Is this common in filesystems? Or is ZFS striking out new ground here? I'm not really an under-the-OS-hood kinda guy.

yjftsjthsd-h 3y ago
> Filesystem-level de-duplication is scary as hell as a concept What's scary about it? You have to track references, but it doesn't seem that hard compared to everything else going on in ZFS et al. > Is this common in filesystems? Or is ZFS striking out new ground here? At least BTRFS does approximately the same.
- cesarb 3y ago
  > > Filesystem-level de-duplication is scary as hell as a concept > What's scary about it? It's scary because there's only one copy when you might have expected two. A single bad block could lose both "copies" at once.
  - ForkMeOnTinder 3y ago
    Copying a file isn't great protection against bad blocks. Modern SSDs, when they fail, tend to fail catastrophically (the whole device dies all at once), rather than dying one block at a time. If you care about the data, back it up on a separate piece of hardware.
  - grepfru_it 3y ago
    The file system metadata is redundant and on a correctly configured ZFS system your error correction is isolated and can be redundant as well
  - Filligree 3y ago
    Disks die all the time anyway. If you want to keep your data, you should have at least two-disk redundancy. In which case bad blocks won't kill anything.

uvatbc 3y ago

Technically, yes: through the use of Truenas that gives us API access to iscsi on ZFS.

deleted 3y ago
[deleted]

rossmohax 3y ago

Does ZFS or any other FS offer special operations which DB engine like RocksDB, SQLite or PostgreSQL could benefit from if they decided to target that FS specifically?

magicalhippo 3y ago
Internally, ZFS is kinda like an object store[1], and there was a project trying to expose the ZFS internals through an object store API rather than through a filesystem API. Sadly I can't seem to find the presentation or recall the name of the project. On the other hand, looking at for example RocksDB[2]: File system operations are not atomic, and are susceptible to inconsistencies in the event of system failure. Even with journaling turned on, file systems do not guarantee consistency on unclean restart. POSIX file system does not support atomic batching of operations either. Hence, it is not possible to rely on metadata embedded in RocksDB datastore files to reconstruct the last consistent state of the RocksDB on restart. RocksDB has a built-in mechanism to overcome these limitations of POSIX file system [...] ZFS does provide atomic operations internally[1], so if exposed it seems something like RocksDB could take advantage of that and forego all the complexity mentioned above. How much that would help I don't know though, but seems potentially interesting at first glance. [1]: https://youtu.be/MsY-BafQgj4?t=442 https://youtu.be/MsY-BafQgj4?t=442 [2]: https://github.com/facebook/rocksdb/wiki/MANIFEST https://github.com/facebook/rocksdb/wiki/MANIFEST
macdice 3y ago
https://commitfest.postgresql.org/43/4338/ https://commitfest.postgresql.org/43/4338/

ludde 3y ago

A M A Z I N G Have been looking forward to this for years! This is so much better than automatically doing dedup and the RAM overhead that entails. Doing offline/RAM+in memory dedup size optimizations seem like a really good optimization path. In the spirit of also paying only what you use and not the rest. Edit: What's the RAM overhead of this? Is it ~64B per 128kB deduped block or what's the magnitude of things?

mlyle 3y ago
> Edit: What's the RAM overhead of this? Is it ~64B per 128kB deduped block or what's the magnitude of things? No real memory impact. There's a regions table that uses 128k of memory per terabyte of total storage (and may be a bit more in the future). So for your 10 petabyte pool using deduping, you'd better have an extra gigabyte of RAM. But erasing files can potentially be twice as expensive in IOPS, even if not deduped. They try to prevent this.
- ludde 3y ago
  If I manually deduplicate/copy a 1GB file, how many such offset/refcount tuples would be created in RAM? Just one for the whole file, or one per underlying 128kB block?
  - rincebrain 3y ago
    n.b. I am not pjd, this is from memory and may be wrong. The answer to that is messy, but basically, there's a table that should be kept in memory for "is there anything reflinked at all in this logical range on disk", and that covers large spans, so how many entries would depend on how contiguous your data logically was on disk; the actual precise mapping list per-vdev doesn't need to be kept continuously in memory, just the more coarse table, so that saves you a fair bit on memory requirements.
  - mlyle 3y ago
    They don't go in ram. You have one 64 bit refcount number for every 64MB of storage irrespective of how much is deduplicated; this is a reduction factor of 8M, or 128k per terabyte stored. This largely exists so you can avoid doing extra work on free.

dark-star 3y ago

Wow, I was under the impression that this had long been implemented already (as it's already in btrfs and other commercial file systems) Awesome!

mgerdts 3y ago
It has been in the Solaris version of zfs for a long time as well. This came a few years after the Oracle-imposed fork. https://blogs.oracle.com/solaris/post/reflink3c-what-is-it-why-do-i-care-and-how-can-i-use-it https://blogs.oracle.com/solaris/post/reflink3c-what-is-it-w...
- rincebrain 3y ago
  reflink is implemented on Oracle Solaris; how it works internally I don't think Oracle has ever commented, since implementing post-facto dedup on a filesystem that assumes data once written is unmodified is a bit Spicy(tm) - you can look at the BRT talk from the OpenZFS dev summit or other talks about it at the leadership meetings to get some idea why it's so exciting.
abwizz 3y ago
implying that zfs is not a commercial filesystem made me chuckle

XorNot 3y ago

Finally! This is exactly what I want for handling photos and media on my home server. We have phone's dumping their camera rolls onto the machine via Syncthing, but the easiest way to share content safely is just to make copies of the file and go "storage is cheap". But the reality has always been closer to "storage should handle deduplication" - with this feature landing that can finally be a reality. EDIT: Not to mention, this should make ZFS best-in-class for handling container filesystems. Deduplicating all the files in the individual layers means you can pretty much dispense with awkwardly trying to maintain inheritance trees.

thinkloop 3y ago
Would you mind expanding on that - what mechanism/app on the phone would share the files?
- XorNot 3y ago
  Syncthing - the android app can share your camera roll folder. So that syncs to my server, and my desktop/laptop. I drag files around on there when I want them deleted off my phone and archived somewhere. Me and my wife share a syncthing folder between us when we want to send files to each other. All of this produces a fair number of duplicate files, particularly if you have backups turned on in case of deletes. Offline dedupe basically makes all of that free though - duplicate files on the server or in backup dirs are no longer a problem.
tecleandor 3y ago
I also was thinking about using this for something similar: organization. I have datasets with unorganized data dumps: photos, music, torrents... Organizing that in shares means duplicating files...

KirillPanov 3y ago

Didn't btrfs have this like a ~decade ago?

LeoPanthera 3y ago
Yes, although be careful, because running "defragment" will un-clone and therefore duplicate all your identical blocks.

tecleandor 3y ago

I also see, VERY interesting: - Linux container support (So overlayfs is ZFS aware) - BLAKE3 checksums (So faster checksums) - Zstd early abort (I though Zstd already had early abort! This means you can enable Zstd compression by default, and if the data isn't compressible, it aborts the operation and stores it uncompressed, avoiding the extra CPU use. - Fully adaptive ARC eviction (Separate adaptive ARC for data and for metadata. Until now metadata ARC was almost FIFO and not efficient) - Prefetch improvements (That's always nice)

someplaceguy 3y ago
> This means you can enable Zstd compression by default, and if the data isn't compressible, it aborts the operation and stores it uncompressed, avoiding the extra CPU use This was already implemented for all compression algorithms since ZFS was released as open source in 2005, at least (and even though zstd didn't yet exist then, it still benefited from this feature when it was integrated into ZFS). The feature you describe only saves CPU time on decompression, but still requires ZFS to try to compress all data before storing it on disk. This new early abort feature actually seems to be even more clever/sophisticated, because it also saves a lot of CPU time when compressing, not just descompressing. Usually you are forced to select a fast compression level so that your performance doesn't degrade too much for little benefit, because if you had a higher compression level, ZFS would always be paying a high CPU cost for trying to compress data which often is not even compressible (since a lot of data on a typical filesystem belongs to large files, which are often not compressible). However, with early abort, you can select a higher compression level (zstd-3 or higher) without paying a penalty for spending a lot of time trying to compress uncompressible data. The way it seems to work is that ZFS first tries to compress your data with LZ4, which is an extremely fast compression algorithm. If LZ4 succeeds in compressing the data, then ZFS discards the LZ4-compressed data and recompresses it with your chosen higher compression level, knowing that it will be worth it. If LZ4 can't compress the data, then ZFS doesn't even try to compress with the slower algorithm, it just stores the data uncompressed. This almost completely avoids paying the CPU penalty for trying to compress data that cannot be compressed, and only results in about 9% less compression savings than if you had simply chosen the higher compression level without early abort. The actual algorithm is even more clever: in-between the LZ4 heuristic and a zstd-3 or higher compression level, there is an intermediate zstd-1 heuristic that is also done to see whether the data is compressible. This intermediate heuristic seems to almost completely eliminate the 9% compression savings cost while still being much faster than trying to compress all data with zstd-3 or higher, since zstd-1 is also very fast. In the end, you get almost all the benefits of a higher compression level without paying almost any cost for wasting CPU cycles trying to compress uncompressible data!

girishso 3y ago

Pardon my ignorance, but isn’t this same as btrfs snapshots? Any advantage of using zfs over btrfs?

abwizz 3y ago
unlikely the same but similar in some aspects. likely zfs and btrfs have some similarities (copy on write) and some differences (integration with volume manager). if you're interested, i'm sure you'll find more.

renewiltord 3y ago

Is the overlayfs/ZFS integration first-class? (i.e. COW on overlayfs is implemented at block layer on ZFS?)