3 ms·
Deduplication requires an absolutely enormous amount of memory, unless you have a dataset that has a huge amount of duplication for some reason (maybe a company
by colechristensen 3y ago
Deduplication requires an absolutely enormous amount of memory, unless you have a dataset that has a huge amount of duplication for some reason (maybe a company or school where a lot of people have their own network storage and put the same large files in?) it is usually never the right choice to enable dedup.
It is cheaper to buy more disk than the memory to enable dedup (somebody could run the numbers). ZFS ARC also likes lots of memory and you get good performance gains with more memory allocated to it.
In other words, turn on LZ4 compression everywhere and don’t think about it otherwise.
- 3np 3y agoThis is my understanding as well. These days we default to zstd, though.
- erhaetherth 3y agoOh good, I have it off. I was pondering that feature years ago when I set it up. Couldn't remember where I landed.
- keep_reading 3y agoDeduplication in ZFS can be pushed onto dedicated flash storage now instead of requiring gobs of RAM.
- ungamedplayer 3y agoGreat channel for side attacks btw.
- nolist_policy 3y agoDoesn't even need SSD's to open a side-channel: https://lore.kernel.org/linux-btrfs/CAKDzk=-HZardsLFH5c9HYre73NYNszUJqpfsh0YJnnaQToB3BA@mail.gmail.com/ https://lore.kernel.org/linux-btrfs/CAKDzk=-HZardsLFH5c9HYre...
- keep_reading 3y agoWithout having physical access to my hardware, where every disk is FDE anyway?
- ungamedplayer 3y agoPhysical access not required. Dedupe side attack required timing measurement.
- ungamedplayer 3y agoOh got some hate there.. pissed off the zfs kids with a dose of reality. Dedupe can be used to infer secret data. .. more news at 11
- CMCDragonkai 3y agoHow to do this?