2 ms·
I think you need to read about unix hard links, since they solve all 5 of your problems at once. The name of each hard link is the hash of the file, and you ar
by ars 17y ago
I think you need to read about unix hard links, since they solve all 5 of your problems at once.
The name of each hard link is the hash of the file, and you are done.
Identical means identical, not close.
Each server will have a copy of attachments for email that lives on that server. No need to do it globally, just de-dup within each server.
I don't know if it saves much, but if your filesystem supports hard links, it's pretty much free.
That said, I don't know how google's GFS does things.
- blasdel 17y agoYou're on the right track, but why the hell do you think your gmail lives on 'a server', much less in a fliesystem directly? GFS normally uses chunks 2^14 times larger than a traditional inode -- you'd use infrastructure built on top of it (like Bigtable) to store email and indexes of it. You could easily end up designing a system that stores the email bodies in Gmail in a content-addressed form just as an indexing method, and end up getting de-duplication for free.