6 ms·
OpenZFS 2.2: Block Cloning, Linux Containers, BLAKE3
- buybackoff 3y agoHaving Proxmox-based homelab setup I was waiting for quite a while for Linux containers support so that Docker in LXC containers just works. It was quite inconvenient when even a pull for a simple image freezes. Hopefully now a docker-composed single logical set of Docker containers (or just a single Docker container) per LXC container will just work without VM overheads.
- throw0101b 3y agoNote that cloning was already possible at the filesystem/snapshot level for a while: * https://openzfs.github.io/openzfs-docs/man/master/8/zfs-clone.8.html https://openzfs.github.io/openzfs-docs/man/master/8/zfs-clon... * https://openzfs.github.io/openzfs-docs/man/master/7/zfsconcepts.7.html#Clones https://openzfs.github.io/openzfs-docs/man/master/7/zfsconce... (ZFS uses the fairly 'standard' nomenclature of "snapshots" being read-only copies and "clones" being read-write copies.)
- seabrookmx 3y ago> support for overlayfs Does this mean I can use the overlay2 driver with docker instead of the ZFS one? I've had many people tell me this was possible in the past when it definitely wasn't. The performance and reliability of the ZFS driver leaves lots to be desired. The last time I ran Docker on a ZFS partition I actually used a zvol formatted ext4 just to avoid these issues.
- pgeorgi 3y ago> The performance and reliability of the ZFS driver leaves lots to be desired. No wonder: it calls the zfs(1) tool and parses its output to do its work. Again. and again. and again.
- LanternLight83 3y agoI once wrote a line of shell piping zfs into awk while walking the tree of datasets, but it was so slow i decided i didn't need to after all. I wonder what a filesystem would look like that made the performance of meta-reflection a primary concern.
- justinclift 3y agoShouldn't it be directly calling the "zed" ZFS daemon to get that info instead?
- pgeorgi 3y agoPerhaps. Thing is, https://github.com/moby/moby/blob/670bc0a46c4ca03b75f1e72f73b8302bce90275a/daemon/graphdriver/zfs/zfs.go https://github.com/moby/moby/blob/670bc0a46c4ca03b75f1e72f73... is using https://github.com/mistifyio/go-zfs https://github.com/mistifyio/go-zfs which features code like `out, err := zfsOutput("get", "-H", key, d.Name)` (Source: https://github.com/mistifyio/go-zfs/blob/master/zfs.go#L315 https://github.com/mistifyio/go-zfs/blob/master/zfs.go#L315) to get a single zfs property. Somebody chose to use a library as abstraction that looks good but is implemented as a MVP (nothing wrong with that). "In the future, we hope to work directly with libzfs" should have raised an alarm somewhere, though.
- justinclift 3y agoInteresting. I'd been meaning to check if there are go libraries for working with ZFS. Looks like the answer to that is "yes, but be careful as some are kind of dodgy". :/ --- Oh, that library bills itself as "Simple wrappers for ZFS command line tools", rather than as a ZFS interface. Wonder why the Moby project picked that one then? Maybe a case of "it was the best choice at the time".
- pgeorgi 3y agoThere's https://github.com/openzfs/zfs/tree/master/contrib/pyzfs https://github.com/openzfs/zfs/tree/master/contrib/pyzfs, which is as official as it gets.
- tgz 3y agoWhat about NVME optimizations?
- dsp_person 3y agoPrevious thread with discussion of the release candidate: https://news.ycombinator.com/item?id=36588240 https://news.ycombinator.com/item?id=36588240 While block cloning isn't supported for encrypted datasets yet, looks like there is a WIP already: https://github.com/openzfs/zfs/pull/14705 https://github.com/openzfs/zfs/pull/14705 Very cool to have the ability for near instant cp or mv between datasets or snapshot (at least locally; the links don't persist with send/recv?).
- v3ss0n 3y agoAny comparison to BTRFS? I am using BTRFS in place of Zfs because it had a problem with Zfs moubts
- candiddevmike 3y agoBoth are CoW filesystems with different strengths and weaknesses. BTRFS is part of the kernel, has interesting integrations with systemd (and possibly better Linux tooling in general), and has more flexibility with adding and removing disks (for now). ZFS has (arguably) better reliability and has more built-in features like encryption, read/write caching, and exposing virtual volumes. I don't think folks should default to using one or the other, everything depends on your workloads. Ext4 might even be the best choice. Though bcachefs could be an "endgame FS" if it delivers on its promises.
- doublepg23 3y agoI’ve come around to using btrfs as basically ext4++ but ZFS is head and shoulders a better product. This article is somewhat outdated but the author is knowledgeable on the subject https://arstechnica.com/gadgets/2021/09/examining-btrfs-linuxs-perpetually-half-finished-filesystem/ https://arstechnica.com/gadgets/2021/09/examining-btrfs-linu...
- Avamander 3y agoZFS doesn't integrate with Linux' page cache well though. That's quite a large negative IMO.
- keep_reading 3y agoBut the reworked ARC implementation in here now will help a lot
- throw0101b 3y ago> Any comparison to BTRFS? ZFS RAID is production quality and safe to use. Not the case with Btrfs: * https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#raid56-status-and-recommended-practices https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#raid5... Certainly striping-mirroring will give you more IOps, but if you want more space efficiency for bulk storage, RAID-y solutions is probably better. ZFS has been in production use since Solaris 10u2 (June 2006), so we're approaching twenty years of it being banged on.
- EvanAnderson 3y agoI'm always impressed w/ ZFS. I really like the look of: > Corrective "zfs receive" (#9372[0]) - A new type of zfs receive which can be used to heal corrupted data in filesystems, snapshots, and clones when a replica of the data already exists in the form of a backup send stream. [0] https://github.com/openzfs/zfs/pull/9372 https://github.com/openzfs/zfs/pull/9372
- XorNot 3y agoBlock cloning is kind of a significant update! I wonder what holds back a ZFS-level offline dedupe function now that that's implemented since you could already basically write a shell script to do something like it.
- simcop2387 3y agoMight be easy to extende fdupes and jdupes to be able to do this without much effort. I haven't seen the api/syscall invokved but I use them with btrfs for a specific use where I have a lot of known duplicates.
- binarycrusader 3y agoFor filesystem geeks (of a sort) like myself, note that ReFS has had block cloning support for some time now: https://learn.microsoft.com/en-us/windows-server/storage/refs/block-cloning https://learn.microsoft.com/en-us/windows-server/storage/ref...
- pseudalopex 3y agoReFS had many problems though.[1] Are they better now? [1] https://www.reddit.com/r/DataHoarder/comments/iow60w/testing_windows_refs_data_integrity_bit_rot/ https://www.reddit.com/r/DataHoarder/comments/iow60w/testing...
- ago 3y agoI tried it a few months ago and ReFS ate my data. No indication of why in event logs or SMART data. It had IsPowerProtected set because I have a UPS and I had a unclean restart, I would expect it to lose data, but not to corrupt the filesystem metadata. I had a backup of the data but wanted some recent changes. Refsutil (the official Microsoft tool) didn't help because it has not been updated for the newest ReFS version. I couldn't read most files because I had integrity enable and files failed the check. Hetman's Data Recovery was able to recover most of the data. In later testing I found out that IsPowerProtected is just very unsafe. I have since put some time into testing and sometimes fixing https://github.com/openzfsonwindows/openzfs https://github.com/openzfsonwindows/openzfs , it is not ready for use yet, but it is making great progress.
- kreetx 3y agoI've recently tried ZFS on an NVMe drive in a USB enclosure, the USB connection is fragile and when rsyncing data onto it, it dis- and reconnects, after which the zpool is in SUSPENDED state and only a reboot helps :(.
- doublepg23 3y agoThis is a very Stack Overflow response but, yes, don’t use USB for any storage you expect to be reliable.
- admax88qqq 3y agoLol, ZFS is supposed to be this super reliable filesystem that does your laundry. But if you hard drive disconnects for a moment tough luck.
- esoterae 3y agoLungs are usually pretty reliable, but they're still working on a long-term fix for when air becomes unavailable.
- throw0101b 3y agoZFS is a very reliable file system. At some point ZFS will give up and say "I cannot reliably save bits." It is your hardware where the unreliability lies and ZFS detects that. Would you rather prefer silent corruption?
- kreetx 3y agoTo be clear, I don't mind it getting suspended when there is a hardware failure. What I don't understand is why aren't there sane way (i.e some command sequence) to re-attach it. Otherwise the feature set seems great: both a volume manager and a file system, seamless compression, deduplication, COW and snapshots - what's not to love :)
- 3y ago
- gavinhoward 3y agoQuestion for ZFS experts: the docs say ([1]) that I can change the checksum used, and I want to change it to BLAKE3. However, based on [2], it seems the checksum doesn't store the type of checksum, so it seems that it's an all-or-nothing thing instead of a setting that only applies to new blocks. Can I actually change the checksum? What happens when I do? Edit: It appears the type of checksum actually is stored per-block, in `blk_prop` ([3]) of `blkptr_t` ([4]). Edit 2: The manpage says that the change only applies to new data, so yes, it's safe. [1]: https://openzfs.github.io/openzfs-docs/Basic%20Concepts/Checksums.html https://openzfs.github.io/openzfs-docs/Basic%20Concepts/Chec... [2]: https://people.freebsd.org/~gibbs/zfs_doxygenation/html/d9/dce/structzio__cksum.html https://people.freebsd.org/~gibbs/zfs_doxygenation/html/d9/d... [3]: https://github.com/openzfs/zfs/blob/c0e58995e33479a9c1d97fb2a19f8f507cc954b7/include/sys/spa.h#L442-L448 https://github.com/openzfs/zfs/blob/c0e58995e33479a9c1d97fb2... [4]: https://github.com/openzfs/zfs/blob/c0e58995e33479a9c1d97fb2a19f8f507cc954b7/include/sys/spa.h#L375-L383 https://github.com/openzfs/zfs/blob/c0e58995e33479a9c1d97fb2...
- throw0101b 3y agoThe checksum type for the block is defined elsewhere (include/sys/zio.h) per a "enum zio_checksum".
- GrilledChips 3y agoIt would go against a fundamental principle of ZFS' design for changing a setting like that mid-flight to not be safe.
- xoa 3y agoA lot of really nice, focused polish items in this release that have real practical benefits. The corrective "healing" receive for example, ZFS has always been able self-heal when there is redundancy (either device level or using the copies attribute), and that's always been a core feature. And in its original incarnation there'd always be at least some redundancy. But as it gets wider use fact is the vast majority of systems have a single device and don't want to cut available storage in half for copies=1. Corruption can still be detected with a scrub thanks to checksums, so ZFS is still useful anyway. It's better to know about corruption asap, while you still have good backups. But being able to smoothly and very quickly repair that without any local redundancy using backups you should have anyway is a nice bit of polish and leveraging of the foundation. One question if anyone who follows it more closely knows: are there any efforts to work specifically on ZVOL performance? They're very handy for iSCSI and other features, but my understanding is they haven't gotten much focus for quite awhile now. Maybe that just doesn't have any real company backing right now for R&D but I hope it gets some attention eventually.
- brnt 3y ago> But as it gets wider use fact is the vast majority of systems have a single device and don't want to cut available storage in half for copies=1 I'm still patiently holding out for the first (current, usable) filesystem that let's us set a parity fraction and let me use FEC to heal small errors.
- xoa 3y agoI suppose in principle you could partition a drive n ways the run RAIDZ1? Though performance would probably be mediocre and there'd be some oddities. Occurs to me know that as a practical matter perhaps raw storage will solve this for a lot of people. The pace of increase in storage/$ still seems to be steady, and in turn continues to mean more and more people trivially have their total needs exceeded. At some point may be perfectly reasonable to throw half at redundancy?
- GrilledChips 3y ago
- dang 3y agoRecent and related: ZFS 2.2.0 (RC): Block Cloning merged - https://news.ycombinator.com/item?id=36588240 https://news.ycombinator.com/item?id=36588240 - July 2023 (165 comments)
- MerelyMortal 3y agoCool, but I'm really looking forward to being able to add disks to an existing volume.
- mrlonglong 3y agoI want this too.
- __MatrixMan__ 3y agoI often find myself wishing for this block cloning thing, but with rsync hooks. Like suppose you're copying a large genome from A to B, and you already have a genome from that species (but different organism) lying around on B. Sure, you could clone and then rsync on top of the clone to avoid transferring the common bits again. But that's forethought that users often don't have. Better to use the rolling hash related metadata that rsync generates as a query into all possible targets and pick one automatically so that the user doesn't have to think about it and just sees it as a really fast copy.
- hotpotamus 3y agoCouldn't you just send a diff/patch?
- __MatrixMan__ 3y agoThat would require you to know ahead of time what you're patching, and to have a copy of it on the source machine so you could generate the patch. This would be automatic. If there's similarity on the target, use it, but without requiring the user to tell you where to find it.
- mrlonglong 3y agoDoes this also include the ability to add additional drives to a zpool? For example, going from 4 disks in to 6 disks with raidz2/raidz3 zpool?
- justinclift 3y agoNah, the PR for that is still being reviewed prior to getting merged: https://github.com/openzfs/zfs/pull/15022 https://github.com/openzfs/zfs/pull/15022
- mrlonglong 3y agoNice, thanks. Looks like it'll be soon. Can drives be removed in the same way if there's the space for it?
- justinclift 3y agoGood question, I'm not personally sure.
- rubatuga 3y agoPretty exciting that we're getting COW support in ZFS. It used to be one of the key features of XFS, but I can stick to ZFS only now.