4 ms·
I read all the comments posted here thus far, and the one thing I don't see is a concern that de-duping, if not done AT the filesystem, would be prohibitively s
by bmelton 17y ago
I read all the comments posted here thus far, and the one thing I don't see is a concern that de-duping, if not done AT the filesystem, would be prohibitively slow.
Am I misguided in thinking that it would be? I mean, I suppose if you implemented it on disk at the server level, retrieving the duped blocks there, I think you're adding latency. I suppose I might be looking at it from the wrong perspective, as there just has to be people using S3 for backups only, for which cost probably does make more sense than speed, but I honestly don't know if that's more the exception or the rule.
- houseabsolute 17y agoThat depends on the block size. If you're using 40MB blocks, for example, then seek costs are not going to dominate, and you can afford to stitch together a couple of deduped blocks. Where you're going to find duplicated files that are large enough that blocks so large make sense is another question.