4 ms·
I'm a long-time ZFS fan, and still for storage of my digital artifacts I don't really trust anything else. For backup servers, where performance isn't really c
by linsomniac 5y ago
I'm a long-time ZFS fan, and still for storage of my digital artifacts I don't really trust anything else. For backup servers, where performance isn't really critical, and snapshots are a huge benefit, it is great!
However, ~2 years ago I installed my laptop with encrypted ZFS, and frankly it kind of sucks. I don't know if this is the SIMD thing, or something else, but I can basically count on 1 full CPU running some zfs process at all times. Any apt install takes a stupid long time because it's doing something with snapshots. And I think I've only used the snapshots a couple of times. When I did though, it was handy.
But, I can't complain about my track record with ZFS: Over ~15 years I've never had data lost by it, despite having some horrible things happen to the systems that have run it.
I will fully agree that dedup is effectively useless. I've never had a system that had enough memory to reliably run dedup on any real workload.
I'll disagree that there are other tools better at doing what ZFS does though. In particular, I've had 5 events over the last year where our (normally reliable) PERC storage systems experienced corruption on LVM and RAID-6 arrays. I believe it was related to some issues with the Dell drives and Kafka broker activity on the arrays, but it sure would have been nice to have had ZFS on them instead of hardware RAID+LVM.
- donmcronald 5y ago> I don't know if this is the SIMD thing, or something else, but I can basically count on 1 full CPU running some zfs process at all times. Based on my (limited) experience it was really bad before the whole SIMD thing got resolved. There was a huge improvement once they worked around that.
- colordrops 5y agoI installed NixOS from scratch on my laptop a couple weeks ago, and went with encrypted ZFS. Not seeing any CPU utilization it.
- Filligree 5y agoI don't know if it ended up implemented, but there was talk of patching Linux to re-enable the necessary symbols for ZFS. This would let it use SIMD again, and speed it up to where encryption isn't a performance hog. It'd explain why you get different results on NixOS. EDIT: Seems like it was, but has been removed as unnecessary on newer ZFS versions.
- linsomniac 5y agoNot sure what the deal is, I haven't really dug into it at all, but pretty much any time I run htop I can see a 100% CPU process like: 2710046 root 20 0 1121M 85744 6848 S 102. 0.3 0:05.76 /sbin/zsysd
- rincebrain 5y agozsysd is an Ubuntu daemon for managing ZFS things, not something OpenZFS provides. So if that is the only process spinning, that is probably specific to them. (Not that OZFS native encryption doesn't have flaws; I am probably the last person on the planet to pick to argue that point. But I don't know that this is among them.)
- exikyut 5y agoI wonder what `sudo perf top` says it's doing (in terms of heaviest symbols). That would probably be the most useful thing to stick on the end of a pitchfork :P (The command accepts a `-p <pid>` parameter; without that it'll profile the whole system (I make this sound much heavier than it is).)
- E39M5S62 5y agozsysd is Ubuntu's zsys tool. It's notoriously bad at burning CPU as your number of snapshots increase. It has some pretty serious flaws as well that can result in total data loss on the local drive due to aggressive / incorrect snapshot pruning.
- pmarreck 5y agoCouldn't you do passive dedup somehow by having a periodic task that searched for dupes and replaced them with hardlinks?
- XorNot 5y agoThis removes copy-on-write though, which is why it's a bad idea. Editing a file in one place and having it surprise edit everywhere else is a good way to wind up with a disaster.
- linsomniac 5y agoI don't think so, for a few reasons: - My primary need for it is on my backup box, which does "rsync --inplace" to keep small changes to large files from completely creating a new copy of that file every backup run. Hardlinks would cause all of those to compete if some previously similar file started getting updated (OS updates or similar starting files that then diverge. - Each system gets it's own ZFS, so I can snapshot them at the end of the backup, keep different retention times, etc. But I can't hardlink across filesystems, I don't think. The linked article did mention increasing dedup block size, which might help. If I could even cut down the DDT by half, that would make it more doable.
- yjftsjthsd-h 5y agoWouldn't the better option be to overwrite one duplicate with `cp --reflink`? That way you keep the nice CoW properties... I think.
- Dylan16807 5y agoZFS can't do reflinks.
- yjftsjthsd-h 5y agoOh, I didn't realize that, thanks
- rincebrain 5y agoComing soon to an OpenZFS near you: BRT [1]. Not inline dedup, but a number of the nicer benefits, and usually cheaper. [1] - https://drive.google.com/file/d/1csE8OuPotfhaFi9KvTGKMGy86KxrBu2W/view?usp=sharing https://drive.google.com/file/d/1csE8OuPotfhaFi9KvTGKMGy86Kx...
- abrookewood 5y agoThat sounds promising. I made the mistake of using de-dupe on ZFS before I understood the RAM requirements and performance implications ... quickly backed out.
- fuzzy2 5y agoFor security reasons I recently encrypted some of my ZFS home server (including the OS SSD). Throughput took a severe hit, because the CPU (Pentium G630) does not support AES-NI. However, day-to-day operations (also CPU load over time) remain virtually unchanged. Encryption is unrelated to any snapshotting activity. Sounds like you may have additional hooks in place. ZFS by itself does not integrate in any way with package management or anything else, really.
- linsomniac 5y agoYes, Ubuntu includes some hooks in the package management for snapshots related to installing packages. I haven't had need to use them so far.