5 ms·

Regarding wasting storage on crud, I would be very surprised if the gmail storage system does not duplicate large attachments that have been forwarded around.

by marze 17y ago

Regarding wasting storage on crud, I would be very surprised if the gmail storage system does not duplicate large attachments that have been forwarded around. It would not be difficult to store one copy of every attachment or identical newsletter and link to it. This has to decrease their storage requirements dramatically.

deleted 17y ago
[deleted]
ilamont 17y ago
It's called "deduplication", and is used in some enterprise networks to free up storage space.
patio11 17y ago
Off the top of my head: 1) What if the disk your canonical copy is on dies? 2) You just introduced a dependency for cross-account reference counting (Alice and Bob share CryptoBible.doc attachment, Alice deletes the message it is attached to, Bob better still be able to access it). Your programmers will love you for that one. Don't forget all sorts of fun edge cases like "Does a suspended account peg an attachment on disk for forever, on the theory they could be unsuspended?" 3) Define "identical newsletter". No, really, pretend this is a job interview question. waits Did you take into account the headers? waits How about the salutation (Dear $FIRST $LAST,)? waits How about the tracking pixels and unsubscribe links, which are by necessity personalized? 4) What is the performance impact of the algorithm to determine "identical", with respect to your kinda-sorta-not-really "identical" developed in question #3? Can it classify billions of messages in realtime to make nigh-instantaneous retain-or-delete-and-add-pointer decisions? 5) Does this algorithm introduce multiple single-points-of-failure into Gmail to save a fraction of a sliver of the costs of running the service?
- ars 17y ago
  I think you need to read about unix hard links, since they solve all 5 of your problems at once. The name of each hard link is the hash of the file, and you are done. Identical means identical, not close. Each server will have a copy of attachments for email that lives on that server. No need to do it globally, just de-dup within each server. I don't know if it saves much, but if your filesystem supports hard links, it's pretty much free. That said, I don't know how google's GFS does things.
  - blasdel 17y ago
    You're on the right track, but why the hell do you think your gmail lives on 'a server', much less in a fliesystem directly? GFS normally uses chunks 2^14 times larger than a traditional inode -- you'd use infrastructure built on top of it (like Bigtable) to store email and indexes of it. You could easily end up designing a system that stores the email bodies in Gmail in a content-addressed form just as an indexing method, and end up getting de-duplication for free.
- blasdel 17y ago
  1) Why do you think Google has the canonical copy on a disk? 2) Why do you think Google ever intentionally deletes anything? Concurrent deletes are hard and expensive no matter what, and all of Google's tools make it much more so. They've gotten in all kinds of trouble in the EU for not being able to provide guarantees about how long they retain data. 3) Store headers, body, and MIME chunks separately. Your datastore will make hashes for its own reasons, and can make the de-duplication decisions independently of your app. 4) Your datastore is using hashes as the keys for retrieving the data already. 5) No, because there is no algorithm for this feature -- you got it for free because of other architectural decisions.
  - pyre 17y ago
    > They've gotten in all kinds of trouble in the EU for not being able to provide guarantees about how long they retain data. The thing that I don't get is why they can't just 'wipe' the data. Maybe I'm misunderstanding GFS/BigTable here, but you don't necessarily need to 'remove' the information from that database. Just 'zero out' the data in there (or overwrite it with random data).
    - blasdel 17y ago
      For starters, there are tons of indexes and snippets extrapolated from the data, and they're cached all over the place. More fundamentally, why do you think any of the records are mutable at all? You reap enormous benefits at that kind of scale by making all writes log-structured.
pyre 17y ago
IIRC, MS Exchange does this. At least w/ attachments. I don't know about attachments from outside of the 'Exchange Ecosystem,' but if one Exchange user sends an attachment to 3 others, only one copy is stored in the Exchange DB.

by marze 17y ago

deleted 17y ago

[deleted]

ilamont 17y ago

It's called "deduplication", and is used in some enterprise networks to free up storage space.

patio11 17y ago

Off the top of my head: 1) What if the disk your canonical copy is on dies? 2) You just introduced a dependency for cross-account reference counting (Alice and Bob share CryptoBible.doc attachment, Alice deletes the message it is attached to, Bob better still be able to access it). Your programmers will love you for that one. Don't forget all sorts of fun edge cases like "Does a suspended account peg an attachment on disk for forever, on the theory they could be unsuspended?" 3) Define "identical newsletter". No, really, pretend this is a job interview question. waits Did you take into account the headers? waits How about the salutation (Dear $FIRST $LAST,)? waits How about the tracking pixels and unsubscribe links, which are by necessity personalized? 4) What is the performance impact of the algorithm to determine "identical", with respect to your kinda-sorta-not-really "identical" developed in question #3? Can it classify billions of messages in realtime to make nigh-instantaneous retain-or-delete-and-add-pointer decisions? 5) Does this algorithm introduce multiple single-points-of-failure into Gmail to save a fraction of a sliver of the costs of running the service?

ars 17y ago
I think you need to read about unix hard links, since they solve all 5 of your problems at once. The name of each hard link is the hash of the file, and you are done. Identical means identical, not close. Each server will have a copy of attachments for email that lives on that server. No need to do it globally, just de-dup within each server. I don't know if it saves much, but if your filesystem supports hard links, it's pretty much free. That said, I don't know how google's GFS does things.
- blasdel 17y ago
  You're on the right track, but why the hell do you think your gmail lives on 'a server', much less in a fliesystem directly? GFS normally uses chunks 2^14 times larger than a traditional inode -- you'd use infrastructure built on top of it (like Bigtable) to store email and indexes of it. You could easily end up designing a system that stores the email bodies in Gmail in a content-addressed form just as an indexing method, and end up getting de-duplication for free.
blasdel 17y ago
1) Why do you think Google has the canonical copy on a disk? 2) Why do you think Google ever intentionally deletes anything? Concurrent deletes are hard and expensive no matter what, and all of Google's tools make it much more so. They've gotten in all kinds of trouble in the EU for not being able to provide guarantees about how long they retain data. 3) Store headers, body, and MIME chunks separately. Your datastore will make hashes for its own reasons, and can make the de-duplication decisions independently of your app. 4) Your datastore is using hashes as the keys for retrieving the data already. 5) No, because there is no algorithm for this feature -- you got it for free because of other architectural decisions.
- pyre 17y ago
  > They've gotten in all kinds of trouble in the EU for not being able to provide guarantees about how long they retain data. The thing that I don't get is why they can't just 'wipe' the data. Maybe I'm misunderstanding GFS/BigTable here, but you don't necessarily need to 'remove' the information from that database. Just 'zero out' the data in there (or overwrite it with random data).
  - blasdel 17y ago
    For starters, there are tons of indexes and snippets extrapolated from the data, and they're cached all over the place. More fundamentally, why do you think any of the records are mutable at all? You reap enormous benefits at that kind of scale by making all writes log-structured.

pyre 17y ago

IIRC, MS Exchange does this. At least w/ attachments. I don't know about attachments from outside of the 'Exchange Ecosystem,' but if one Exchange user sends an attachment to 3 others, only one copy is stored in the Exchange DB.