10 ms·
How to handle big repositories with Git
- CJefferson 12y agoThis is very disappointing when it comes to dealing with large files. It looks like git-bigfiles was abandoned years ago, after making very little progress, and neither bitbucket nor github seem to usefully support git-annex.
- jradd 12y agoI cannot tell if I am way off topic here, so pardon my attempt to hijack your thread, but does anybody know of a solid solution in this respect? Or even a good way to just simply sync and manage larger datasets or files over www. I am not so concerned about version tracking/management so much as just a good way to sync large amounts of data while having a git–like CLi arsenal. Or a comparable CLi solution, or API with decent version tracking and solid support. I have mounted google drive as a linux volume which is buggy to say the least. I have used github, and local git repos, but I am wondering if I am missing out on anything. For example, BitTorrent has a Sync tool that seems close to what I want.
- taeric 12y agoI haven't used it, but https://git-annex.branchable.com/ https://git-annex.branchable.com/ sounds like it fits your bill rather nicely.
- jefurii 12y agoI'm currently working on a geographically distributed digital archives project. We have hundreds of collections of large binary files (video, audio, archival-quality scans) and their associated textual metadata. We're using Git and git-annex to manage all these files and sync them between locations. Some of the collection repositories are pretty big. One collection has 21000 5MB JPGs. Another has 283 190MB files. Some operations do indeed take time, but you expect that when you're working with so many files. git-annex is awesome!
- jradd 12y agoLate response here, but I will have to try this one out as well. git-annex was [virtually] the only result returned from my breif google search. Weird that P4 was no where to be found. Going to have to try both of them out once I get google fiber. :)
- montecarl 12y agoI don't have an answer, but I have an additional complaint. Git itself is pretty terrible at dealing with large repositories if for no other reason than because "git clone" has no resume feature. If your internet connection is interrupted at any point during the initial clone it dies and you have to start over. This means that on an unreliable internet connection it is nearly impossible to clone large repositories.
- jerf 12y agoA git clone is not the same as just rsyncing a .git directory, but if you know Git, the latter can be made to work. You may want to GC the source repo first, and you'll need to manually add a remote and possibly clean up branches after you're done but a quick bash script can finish that up a treat too. (Good task to practice bash scripting on, if you're not familiar with it.)
- bronson 12y agoI've done that, waste of time. There's just no excuse for git being unable to resume a clone.
- chrismonsanto 12y agoI've never heard this complaint before. I would have never guessed it would be a problem. Anyway, for the clone, you could just use a different command (such as rsync) to download the other Git repository to your HD, and use git clone on the local path. There is little difference since git clone grabs everything. Subsequent pulls can be done directly to the remote repository.
- shoo 12y agoi've been using unison to sync a checked-out source repository between a host linux machine and a windows vm, over ssh. if you only want uni-directional synchronisation, then maybe just use rsync? this is distinct from your use case as i'm not using it to sync with remote machines, but it might be worth checking out. http://www.cis.upenn.edu/~bcpierce/unison/ http://www.cis.upenn.edu/~bcpierce/unison/
- jradd 12y agoThanks shoo. I wish this thread wasn't dead so that maybe someone would respond that has used unison. Will check it out if I don't end up loving P4 or git-annex :)
- Igglyboo 12y agoMost game design companies use Perforce specifically for this reason, they have massive asset files that also need to be version controlled and Perforce usually does this better than other choices such as git/hg/svn. Sadly Perforce is proprietary and quite expensive, but it's basically the gold standard in game dev.
- forrestthewoods 12y agoIf you need source control and large binary files then use Perforce. It's pretty solved.
- jradd 12y agoThank you!
- jephir 12y agoI've used submodules before to separate game assets and it just ended up being more trouble than it's worth. The problem is that it separates the history between your two repositories so it takes more time to figure out which asset commit relates to which code commit. In the end, we just merged everything back into one repository.
- eropple 12y agoAgreed. I have a but of a hack that I've stayed to use - I've been using SVN for my game assets and I have git hooks to pull the correct version. Still not optimal, but better.
- jordigh 12y agoI have been quite happy with Mercurial's largefile extension, which has been part of core hg for quite some time: http://mercurial.selenic.com/wiki/LargefilesExtension http://mercurial.selenic.com/wiki/LargefilesExtension I think this must be one of the reasons why hg has some popularity in gamedev.
- shoo 12y agosome of these problems / symptoms can be framed as a dependency-management problem, which can be addressed by other tools: http://blogs.atlassian.com/2014/04/git-project-dependencies/ http://blogs.atlassian.com/2014/04/git-project-dependencies/ that said, it's not like that doesn't introduce other problems
- hackerpolicy 12y agoWould Android's `repo` be a good solution to this?
- fishywang 12y agoYes and No. Repo was born to resolve this kind of problem, but it might not be a "good" one. We (git/gerrit team at Google) are working on bring cross-repository atomic submit and other stuff to git/gerrit and our goal is to replace repo with git submodule. Here're a very brief slides[1] and notes[2] about this topic at this year's Gerrit User Summit. [1] https://docs.google.com/presentation/d/1qG1eAiDmyozZBiVE6R4AVqaZQGRjSUldsDkvFW8AMzA/edit#slide=id.g1d6bea5f2_0134 https://docs.google.com/presentation/d/1qG1eAiDmyozZBiVE6R4A... [2] https://docs.google.com/document/d/1a2eFhVr1HUiKOjhaRHn_89mfnDAIsfpEiHd858ks8vs/edit https://docs.google.com/document/d/1a2eFhVr1HUiKOjhaRHn_89mf...
- ibrahima 12y agoThat's very interesting, as someone who occasionally contributes to the odd Android fork. I kind of feel like repo is this weird semi-black box which does things I don't expect, like reverting my topic branches back to the remote branch but leaving the name in place (it does this if your topic branch does not have a remote tracking branch, and you run repo sync). I feel that at a minimum repo should play nice with standard git workflows. I should probably just read the repo source but I can't imagine why this behavior would be a good idea, and I'm usually left feeling like I would rather just use git directly than repo most of the time.
- jayvanguard 12y ago> Even though the bounds that identify a repository as massive are pretty high – for example the latest Linux kernel clocks at 15+ million lines of code Yeah 15 million lines of code isn't a massive repository, it is medium-large at best. Any one of the big enterprise software companies has repos an order of magnitude bigger for each major product they sell.
- jonalmeida 12y agoDidn't Linus mention this in his talk at Google, Mercurial offers similar speeds, but Perforce would be the option for large binary files. I personally haven't come across any need to use anything except shallow clones in large repos. Most of the time, you want to keep those other topic branches regardless. They've linked to their previous most regarding submodule in the post[1], but it's worth re-mentioning that if you need to use submodule, you should almost always use subtree. [1]: http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree/ http://blogs.atlassian.com/2013/05/alternatives-to-git-submo...
- theli0nheart 12y agoI wrote a Git extension about a year ago that transparently stores data in S3 / Cloudfiles / etc. and doesn't store any of the actual data in your Git repo. I've used it with a few projects but I think it could be battle tested a bit more. It integrates perfectly with GitHub / Bitbucket. Pull requests welcome! https://github.com/lionheart/git-bigstore https://github.com/lionheart/git-bigstore
- caust1c 12y agoI'll throw in a word for git-fat[1]. It works great and has a dead simple implementation that actually has a real copy of the file in the working copy of the repository (as a posed to symlinks). https://github.com/cyaninc/git-fat https://github.com/cyaninc/git-fat
- greggman 12y agoVersion control is not enough for binary files. Binary files need access control since changes can not be merged. Older centralized version control often provides access control as well. Distributed version control can't do this by it's very nature
- jordigh 12y agoLike they say, there's an extension for that: http://mercurial.selenic.com/wiki/LockExtension http://mercurial.selenic.com/wiki/LockExtension
- greggman 12y agoUnfortunately by definition that's not a solution. Because hg is distributed checking out the file locally does nothing unless the fact that it's locked is propagated to everyone else's copy of the repo immediately and that doesn't happen.
- jordigh 12y agoBy definition? Did you even see what this does?
- to3m 12y agoI'm not sure why this ever got downvoted, because it's a key point. Unless you've got some kind of exclusive check out/check in type facility, working with unmergeable binary files is going to prove a pain. It's no fun having people find out once they come to commit/check in/merge/etc. that they've got the unenviable choice of losing their work, or losing somebody else's... or losing both. (The disk space problem is far less of an issue - disk space is cheap, and networks are pretty fast, so up to some medium repository size of (I suppose) hundreds of GB you can actually just let it build up. This doesn't scale completely, but for many projects, possibly even most of them, if everybody has a copy of absolutely everything, it's no problem.) For whatever reason DVCS users often have a bit of a blind spot in this regard. Because DVCSes are good, the thinking often seems to run, if DVCSes handle something poorly, that must be something that revision control simply isn't to be used for. ("You must be using it wrong. Try storing everything on a shared drive and take daily backups and coordinate access using physical tokens.") Which is wrong, because revision control is for everybody, not just the programmers. People who work on unmergeable binary files need it too. But fortunately, even if git/etc. and their users doesn't care about you, you're not completely stuck. if you have lots of binary files your needs are fairly well served by Perforce, and you can probably get on OK with SVN and its locking facility.
- TheJamie 12y agolol @ handling big repositories