Deduplicating filesystems do exist; ZFS is one of the better examples. Of course, maintaining the indexes turns out to become a larger percentage of your cpu ti

Deduplicating filesystems do exist; ZFS is one of the better examples. Of course, maintaining the indexes turns out to become a larger percentage of your cpu time, and those indexes have to live somewhere so they occupy a lot of memory. (You could reduce that impact by doing file-level rather than block -level deduplication, but then you also lose many of the benefits) If you want something that's higher level, there are tools like git annex that do this.