3 ms·
> I don't know what's the median size of your objects but if it's < 20k... I can't remember the exact number off the top of my head, but I designed the average
by rlpb 12y ago
> I don't know what's the median size of your objects but if it's < 20k...
I can't remember the exact number off the top of my head, but I designed the average size of each object to be much bigger - more like 64-256M than kilobytes. IMHO this works far better for backups. So I just use the filesystem to store the blobs, which I think works better.
- rakoo 12y agoDoesn't that alleviate the benefit of deduplication, if you're working on multi megabytes objects ? You'll end up copying a lot, unless I'm missing something obvious...
- rlpb 12y agoMost backups have very large areas of duplication. If a small file has changed, chances are that the small files around it have changed also. So de-duplicating with a larger chunk size seems to work fine in practice.
- rlpb 12y agoCorrection: I'd forgotten the details. Looks like I aimed for 256k, and it is this size that works well. I did consider filesystem performance when I chose the size, intending flat file blob storage here.