3 ms·
> whether or not the filesystem attempts deduplication Unless I'm just parroting truisms here, dedupe is almost never a good idea and compression is a much mor
by 3np 3y ago
> whether or not the filesystem attempts deduplication
Unless I'm just parroting truisms here, dedupe is almost never a good idea and compression is a much more common case-by-case.
> So personally I'd encourage messing around (on something non-prod, of course). Measure measure measure! Does it actually make any difference.
100%. So much depends on the particular workload and resources.
> And making backups, even for ZFS backed filesystems. Sacrilege, I know.
Logical backups, specifically. We have seen more than once corruption and data-loss in replicated encrypted snapshots due to bugs in ZFS.
- colechristensen 3y agoDeduplication requires an absolutely enormous amount of memory, unless you have a dataset that has a huge amount of duplication for some reason (maybe a company or school where a lot of people have their own network storage and put the same large files in?) it is usually never the right choice to enable dedup. It is cheaper to buy more disk than the memory to enable dedup (somebody could run the numbers). ZFS ARC also likes lots of memory and you get good performance gains with more memory allocated to it. In other words, turn on LZ4 compression everywhere and don’t think about it otherwise.
- 3np 3y agoThis is my understanding as well. These days we default to zstd, though.
- erhaetherth 3y agoOh good, I have it off. I was pondering that feature years ago when I set it up. Couldn't remember where I landed.
- keep_reading 3y agoDeduplication in ZFS can be pushed onto dedicated flash storage now instead of requiring gobs of RAM.
- ungamedplayer 3y agoGreat channel for side attacks btw.
- nolist_policy 3y agoDoesn't even need SSD's to open a side-channel: https://lore.kernel.org/linux-btrfs/CAKDzk=-HZardsLFH5c9HYre73NYNszUJqpfsh0YJnnaQToB3BA@mail.gmail.com/ https://lore.kernel.org/linux-btrfs/CAKDzk=-HZardsLFH5c9HYre...
- keep_reading 3y agoWithout having physical access to my hardware, where every disk is FDE anyway?
- ungamedplayer 3y agoPhysical access not required. Dedupe side attack required timing measurement.
- ungamedplayer 3y agoOh got some hate there.. pissed off the zfs kids with a dose of reality. Dedupe can be used to infer secret data. .. more news at 11
- CMCDragonkai 3y agoHow to do this?
- eternityforest 3y agoData loss bugs in ZFS?? Wow. My personal approach to filesystem evaluation is pretty much "How long has it been since I've seen someone talk about a data loss bug". This is my first time hearing one for ZFS. BTRFS seems to have at least one post a year. Ext4 has almost none, but maybe that's because people trust it and blame hardware when unsure what the issue is.
- 3np 3y agoComparing zfs/btrfs with ext4 is not entirely fair. They're in different domains when it comes to scope and features. To some extent, silent corruption in ext4 is expected but in zfs it's a bug.