3 ms·
File system dedupe is expensive because it requires another hash calculation that cannot be shared with application-level hashing, is a relatively rare OS-fs fe
by burnt-resistor 7mo ago
File system dedupe is expensive because it requires another hash calculation that cannot be shared with application-level hashing, is a relatively rare OS-fs feature, doesn't play nice with backups (because files will be duplicated), and doesn't scale across boxes.
A simpler solution is application-level dedupe that doesn't require fs-specific features. Simple scales and wins. And plays nice with backups.
Hash = sha256 of file, and abs filename = {{aa}}/{{bb}}/{{cc}}/{{d}} where
aa = hash 2 hex most significant digits
bb = hash next 2 hex digits
cc = hash next 2 hex after that
d = remaining hex digits
- UltraSane 7mo agoAll good backup software should be able to do deduped incremental backups at the block level. I'm used to veeam and commvault
- burnt-resistor 7mo agoThat costs even more, unreuseable time and effort. It's simpler to dedupe at the application level rather than shift the burden onto N things. I guess you don't understand or appreciate simplicity.
- UltraSane 7mo agoThis article shows it really isn't that simple and is easy to mess up. Who cares if your storage and backup software both dedupe?
- otterley 7mo agoFor ZFS, at least, `zfs send` is the backup solution. And it performs incremental backups with the `-i` argument.
- UltraSane 7mo agozfs send is really awesome when combined with dedupe and incremental