3 ms·
I meant TRANSPARENT filesystem level dedupe. They are doing it at the application level. filesystem level dedupe makes it impossible to store the same file more
by UltraSane 7mo ago
I meant TRANSPARENT filesystem level dedupe. They are doing it at the application level. filesystem level dedupe makes it impossible to store the same file more than once and doesn't consume hardlinks for the references. It is really awesome.
- mmh0000 7mo agoFilesystem/file level dedupe is for suckers. =D If the greatest filesystem in the world were a living being, it would be our God. That filesystem, of course, is ZFS. Handles this correctly: https://www.truenas.com/docs/references/zfsdeduplication/ https://www.truenas.com/docs/references/zfsdeduplication/
- UltraSane 7mo agoI was talking about block level dedupe.
- mmh0000 7mo agoI thought you might be. I just wanted to mention ZFS. Have I mentioned how great ZFS is yet?
- otterley 7mo agoZFS is great! However, it's too complicated for most Linux server use cases (especially with just one block device attached); it's not the default (root filesystem); and it's not supported for at least one major enterprise Linux distro family.
- vmilner 7mo agoIt's not as good as ed: https://www.gnu.org/fun/jokes/ed-msg.html https://www.gnu.org/fun/jokes/ed-msg.html
- burnt-resistor 7mo agoFile system dedupe is expensive because it requires another hash calculation that cannot be shared with application-level hashing, is a relatively rare OS-fs feature, doesn't play nice with backups (because files will be duplicated), and doesn't scale across boxes. A simpler solution is application-level dedupe that doesn't require fs-specific features. Simple scales and wins. And plays nice with backups. Hash = sha256 of file, and abs filename = {{aa}}/{{bb}}/{{cc}}/{{d}} where aa = hash 2 hex most significant digits bb = hash next 2 hex digits cc = hash next 2 hex after that d = remaining hex digits
- UltraSane 7mo agoAll good backup software should be able to do deduped incremental backups at the block level. I'm used to veeam and commvault
- burnt-resistor 7mo agoThat costs even more, unreuseable time and effort. It's simpler to dedupe at the application level rather than shift the burden onto N things. I guess you don't understand or appreciate simplicity.
- UltraSane 7mo agoThis article shows it really isn't that simple and is easy to mess up. Who cares if your storage and backup software both dedupe?
- otterley 7mo agoFor ZFS, at least, `zfs send` is the backup solution. And it performs incremental backups with the `-i` argument.
- UltraSane 7mo agozfs send is really awesome when combined with dedupe and incremental