4 ms·
It's true that git's pack files are made for being rewritten, but bup doesn't do that. Every new run will create a new pack along with its .idx (which means tha
by rakoo 12y ago
It's true that git's pack files are made for being rewritten, but bup doesn't do that. Every new run will create a new pack along with its .idx (which means that some packs may be quasi-empty) and the size of packs is capped at 1GB (Giga, not Gibi).
The real struggle of bup is how to know whether a hash is already stored, and how to know it screaming fast. It could be interesting to compare bup style and standard sqlite style, as you do in ddar.
Also, it seems ddar stores each object in its own file, like git loose objects. SQLite has a page [0] that compares this and storing blobs in SQLite, and I don't know what's the median size of your objects but if it's < 20k it seems to be better to just store them as blobs.
[0] https://www.sqlite.org/intern-v-extern-blob.html https://www.sqlite.org/intern-v-extern-blob.html
- rlpb 12y ago> I don't know what's the median size of your objects but if it's < 20k... I can't remember the exact number off the top of my head, but I designed the average size of each object to be much bigger - more like 64-256M than kilobytes. IMHO this works far better for backups. So I just use the filesystem to store the blobs, which I think works better.
- rakoo 12y agoDoesn't that alleviate the benefit of deduplication, if you're working on multi megabytes objects ? You'll end up copying a lot, unless I'm missing something obvious...
- rlpb 12y agoMost backups have very large areas of duplication. If a small file has changed, chances are that the small files around it have changed also. So de-duplicating with a larger chunk size seems to work fine in practice.
- rlpb 12y agoCorrection: I'd forgotten the details. Looks like I aimed for 256k, and it is this size that works well. I did consider filesystem performance when I chose the size, intending flat file blob storage here.