3 ms·
> You're right that they are doing this at scale. Not in my experience. My product uses >5PB data and we don't bother to do significant deduping. I've only hea
by houseabsolute 17y ago
> You're right that they are doing this at scale.
Not in my experience. My product uses >5PB data and we don't bother to do significant deduping. I've only heard of a few products at Google that do, and that's only when you can anticipate that most objects in the system are going to be duplicated many times, which is decidedly not the case in heterogeneous storage like S3's.