4 ms·
1) Why do you think Google has the canonical copy on a disk? 2) Why do you think Google ever intentionally deletes anything? Concurrent deletes are hard and ex
by blasdel 17y ago
1) Why do you think Google has the canonical copy on a disk?
2) Why do you think Google ever intentionally deletes anything? Concurrent deletes are hard and expensive no matter what, and all of Google's tools make it much more so. They've gotten in all kinds of trouble in the EU for not being able to provide guarantees about how long they retain data.
3) Store headers, body, and MIME chunks separately. Your datastore will make hashes for its own reasons, and can make the de-duplication decisions independently of your app.
4) Your datastore is using hashes as the keys for retrieving the data already.
5) No, because there is no algorithm for this feature -- you got it for free because of other architectural decisions.
- pyre 17y ago> They've gotten in all kinds of trouble in the EU for not being able to provide guarantees about how long they retain data. The thing that I don't get is why they can't just 'wipe' the data. Maybe I'm misunderstanding GFS/BigTable here, but you don't necessarily need to 'remove' the information from that database. Just 'zero out' the data in there (or overwrite it with random data).
- blasdel 17y agoFor starters, there are tons of indexes and snippets extrapolated from the data, and they're cached all over the place. More fundamentally, why do you think any of the records are mutable at all? You reap enormous benefits at that kind of scale by making all writes log-structured.