3 ms·
There are a few things wrong with the git data model, one is lack of chunking, which makes versioning large files space inefficient. Some backup systems like bo
by pabs3 6y ago
There are a few things wrong with the git data model, one is lack of chunking, which makes versioning large files space inefficient. Some backup systems like borg and restic have fixed that issue in the git data model, but suffer from lack of chunking in directory listings, I assume git would have that issue too.
- rakoo 6y agoActually if you want to give credits were it is due, you should mention bup (https://github.com/bup/bup https://github.com/bup/bup) that arguably came up with the idea of content-defined-chunking for backups, and even stores it in git.
- stevesimmons 6y agoSome nice references here are: * Using Rabin fingerprinting to break files into chunks: background here on SO, including link to Rabin's 1981 paper: https://cs.stackexchange.com/questions/82150/how-does-rabin-fingerprint-finds-breakpoints-chunk-boundary https://cs.stackexchange.com/questions/82150/how-does-rabin-... * Andrew Tridgell's PhD thesis on the rsync algorithm (1999): https://rsync.samba.org/~tridge/phd_thesis.pdf https://rsync.samba.org/~tridge/phd_thesis.pdf * Graham Keeling's Masters thesis on content-defined chunking used in his Burp backup program (2014): https://burp.grke.org/images/burp2-report.pdf https://burp.grke.org/images/burp2-report.pdf. Burp dates back to 2011 at least. Section 6 of the thesis has an interesting contemporary comparison of rsync, bup, burp, bacula, rdiff-backup and a few others. * Bup's design docs: https://raw.githubusercontent.com/bup/bup/master/DESIGN https://raw.githubusercontent.com/bup/bup/master/DESIGN * A Linux Weekly News review of Bup from 2010, with comments on strengths and weaknesses of the other approaches then available: https://lwn.net/Articles/380983/ https://lwn.net/Articles/380983/ Digging these out was a trip down memory lane. I tried most of them at various times, and now use borg-backup (https://borgbackup.readthedocs.io https://borgbackup.readthedocs.io).
- globular-toast 6y agoMaybe it's better to say git could be the last SCM then. A VCS is much more general and git certainly won't be the last version control system for things that aren't source code.
- nix23 6y ago>A VCS is much more general and git certainly won't be the last version control system for things that aren't source code. Truer words have never been spoken!!