5 ms·

Ever try committing a 100meg file into git? 100% CPU for 10s of seconds. Game over. Git is a great idea for this, but in practice it performs terribly.

emillon 15y ago
git-annex is a nice extension for this kind of use case. Metadata are stored in git, and file contents are stored separately. http://git-annex.branchable.com/ http://git-annex.branchable.com/
Animus7 15y ago
Sure, if you do this the naive way (map files onto the git repo 1:1). I don't know how this project is doing it, but adding a buffering/sharding layer for large files would alleviate this problem at the cost of worse delta compression (among other things). That said, I still don't think git is necessarily the best platform to base a Dropbox alternative upon, but the commit resource hogging is very fixable.
- hemancuso 15y ago
  It's unclear to me how buffering/sharding could fix this? The bytes still need to get into the repo. That's still expensive and time consuming even if it is chunked or asynchronous.
  - nickik 15y ago
    Look at bup a backup system based on git. They face simular problems. I was at a talk by one of the developers. Very intressting and very technical about how they solve some of the problems, spezally about large files. https://www.youtube.com/watch?v=u_rOi2OVvwU https://www.youtube.com/watch?v=u_rOi2OVvwU https://github.com/apenwarr/bup https://github.com/apenwarr/bup
paulhauggis 15y ago
I have this same problem with mercurial. Is there a better alternative for source control of large files?
- hemancuso 15y ago
  I hear perforce is the go to solution for people who really need this. Or dedicated media asset management solutions.
  - Jach 15y ago
    We recently finished migrating a >6 year old project from perforce to git, splitting out a horrible bloated thirdparty/ directory full of binaries into its own git submodule to eventually be replaced by something sane(r) like Maven or Ivy. I hear subversion is Good Enough for large file dumps, and I'd rather use that through a git interface than its own. p4 and git-p4 are both pretty awful I think after using them. Of course the general response to source control of large files is "don't do that!"
  - forrestthewoods 15y ago
    Perforce is expensive ($740 per user) but I can't imagine using anything else. Just about every major game studio uses it to the best of my knowledge.
  - Corrado 15y ago
    Just be aware that Perforce is a server based system and not really on-par with Git/Mercurial. I'm forced to use it at work and while it works great for storing large binaries (images, movies, etc.) it sucks for daily use, IMHO. At least compared to a modern DVCS.
- spicyj 15y ago
  The extension "kbfiles" may do what you want: http://kiln.stackexchange.com/questions/1873/how-do-you-use-kbfiles http://kiln.stackexchange.com/questions/1873/how-do-you-use-...
  - 15y ago
SimonSapin 15y ago
bup uses the git repository format (packfiles) but splits big files into chuck to avoid this problem. Also does de-duplication at the chunk level. https://github.com/apenwarr/bup/#readme https://github.com/apenwarr/bup/#readme https://github.com/apenwarr/bup/blob/master/DESIGN#L92 https://github.com/apenwarr/bup/blob/master/DESIGN#L92
- nickik 15y ago
  A couple of days ago I attended a talk about bup, might be intressting to you. (really technical) https://www.youtube.com/watch?v=u_rOi2OVvwU https://www.youtube.com/watch?v=u_rOi2OVvwU
Orva 15y ago
It gets only worse when you keep adding more binary files. After doing this for while your repo will not work unless you pack it, which will take horrific times. If it will succeed: my git was 'surprisingly' segfaulting when trying to acquire 16Gb of ram at once during pack operation. (I was trying to keep RAW-photos synced, git stopped working when there was about 4Gb of files stored.) Git is made and optimized for text files, no big surprise that it doesn't cope very well with binaries.

emillon 15y ago

git-annex is a nice extension for this kind of use case. Metadata are stored in git, and file contents are stored separately. http://git-annex.branchable.com/ http://git-annex.branchable.com/

Animus7 15y ago

Sure, if you do this the naive way (map files onto the git repo 1:1). I don't know how this project is doing it, but adding a buffering/sharding layer for large files would alleviate this problem at the cost of worse delta compression (among other things). That said, I still don't think git is necessarily the best platform to base a Dropbox alternative upon, but the commit resource hogging is very fixable.

hemancuso 15y ago
It's unclear to me how buffering/sharding could fix this? The bytes still need to get into the repo. That's still expensive and time consuming even if it is chunked or asynchronous.
- nickik 15y ago
  Look at bup a backup system based on git. They face simular problems. I was at a talk by one of the developers. Very intressting and very technical about how they solve some of the problems, spezally about large files. https://www.youtube.com/watch?v=u_rOi2OVvwU https://www.youtube.com/watch?v=u_rOi2OVvwU https://github.com/apenwarr/bup https://github.com/apenwarr/bup

paulhauggis 15y ago

I have this same problem with mercurial. Is there a better alternative for source control of large files?

hemancuso 15y ago
I hear perforce is the go to solution for people who really need this. Or dedicated media asset management solutions.
- Jach 15y ago
  We recently finished migrating a >6 year old project from perforce to git, splitting out a horrible bloated thirdparty/ directory full of binaries into its own git submodule to eventually be replaced by something sane(r) like Maven or Ivy. I hear subversion is Good Enough for large file dumps, and I'd rather use that through a git interface than its own. p4 and git-p4 are both pretty awful I think after using them. Of course the general response to source control of large files is "don't do that!"
- forrestthewoods 15y ago
  Perforce is expensive ($740 per user) but I can't imagine using anything else. Just about every major game studio uses it to the best of my knowledge.
- Corrado 15y ago
  Just be aware that Perforce is a server based system and not really on-par with Git/Mercurial. I'm forced to use it at work and while it works great for storing large binaries (images, movies, etc.) it sucks for daily use, IMHO. At least compared to a modern DVCS.
spicyj 15y ago
The extension "kbfiles" may do what you want: http://kiln.stackexchange.com/questions/1873/how-do-you-use-kbfiles http://kiln.stackexchange.com/questions/1873/how-do-you-use-...
- 15y ago

SimonSapin 15y ago

bup uses the git repository format (packfiles) but splits big files into chuck to avoid this problem. Also does de-duplication at the chunk level. https://github.com/apenwarr/bup/#readme https://github.com/apenwarr/bup/#readme https://github.com/apenwarr/bup/blob/master/DESIGN#L92 https://github.com/apenwarr/bup/blob/master/DESIGN#L92

nickik 15y ago
A couple of days ago I attended a talk about bup, might be intressting to you. (really technical) https://www.youtube.com/watch?v=u_rOi2OVvwU https://www.youtube.com/watch?v=u_rOi2OVvwU

Orva 15y ago

It gets only worse when you keep adding more binary files. After doing this for while your repo will not work unless you pack it, which will take horrific times. If it will succeed: my git was 'surprisingly' segfaulting when trying to acquire 16Gb of ram at once during pack operation. (I was trying to keep RAW-photos synced, git stopped working when there was about 4Gb of files stored.) Git is made and optimized for text files, no big surprise that it doesn't cope very well with binaries.