5 ms·

I cannot tell if I am way off topic here, so pardon my attempt to hijack your thread, but does anybody know of a solid solution in this respect? Or even a good

by jradd 12y ago

I cannot tell if I am way off topic here, so pardon my attempt to hijack your thread, but does anybody know of a solid solution in this respect? Or even a good way to just simply sync and manage larger datasets or files over www. I am not so concerned about version tracking/management so much as just a good way to sync large amounts of data while having a git–like CLi arsenal. Or a comparable CLi solution, or API with decent version tracking and solid support. I have mounted google drive as a linux volume which is buggy to say the least. I have used github, and local git repos, but I am wondering if I am missing out on anything. For example, BitTorrent has a Sync tool that seems close to what I want.

taeric 12y ago
I haven't used it, but https://git-annex.branchable.com/ https://git-annex.branchable.com/ sounds like it fits your bill rather nicely.
- jefurii 12y ago
  I'm currently working on a geographically distributed digital archives project. We have hundreds of collections of large binary files (video, audio, archival-quality scans) and their associated textual metadata. We're using Git and git-annex to manage all these files and sync them between locations. Some of the collection repositories are pretty big. One collection has 21000 5MB JPGs. Another has 283 190MB files. Some operations do indeed take time, but you expect that when you're working with so many files. git-annex is awesome!
  - jradd 12y ago
    Late response here, but I will have to try this one out as well. git-annex was [virtually] the only result returned from my breif google search. Weird that P4 was no where to be found. Going to have to try both of them out once I get google fiber. :)
montecarl 12y ago
I don't have an answer, but I have an additional complaint. Git itself is pretty terrible at dealing with large repositories if for no other reason than because "git clone" has no resume feature. If your internet connection is interrupted at any point during the initial clone it dies and you have to start over. This means that on an unreliable internet connection it is nearly impossible to clone large repositories.
- jerf 12y ago
  A git clone is not the same as just rsyncing a .git directory, but if you know Git, the latter can be made to work. You may want to GC the source repo first, and you'll need to manually add a remote and possibly clean up branches after you're done but a quick bash script can finish that up a treat too. (Good task to practice bash scripting on, if you're not familiar with it.)
  - bronson 12y ago
    I've done that, waste of time. There's just no excuse for git being unable to resume a clone.
- chrismonsanto 12y ago
  I've never heard this complaint before. I would have never guessed it would be a problem. Anyway, for the clone, you could just use a different command (such as rsync) to download the other Git repository to your HD, and use git clone on the local path. There is little difference since git clone grabs everything. Subsequent pulls can be done directly to the remote repository.
shoo 12y ago
i've been using unison to sync a checked-out source repository between a host linux machine and a windows vm, over ssh. if you only want uni-directional synchronisation, then maybe just use rsync? this is distinct from your use case as i'm not using it to sync with remote machines, but it might be worth checking out. http://www.cis.upenn.edu/~bcpierce/unison/ http://www.cis.upenn.edu/~bcpierce/unison/
- jradd 12y ago
  Thanks shoo. I wish this thread wasn't dead so that maybe someone would respond that has used unison. Will check it out if I don't end up loving P4 or git-annex :)
Igglyboo 12y ago
Most game design companies use Perforce specifically for this reason, they have massive asset files that also need to be version controlled and Perforce usually does this better than other choices such as git/hg/svn. Sadly Perforce is proprietary and quite expensive, but it's basically the gold standard in game dev.
forrestthewoods 12y ago
If you need source control and large binary files then use Perforce. It's pretty solved.
- jradd 12y ago
  Thank you!

by jradd 12y ago

taeric 12y ago

I haven't used it, but https://git-annex.branchable.com/ https://git-annex.branchable.com/ sounds like it fits your bill rather nicely.

jefurii 12y ago
I'm currently working on a geographically distributed digital archives project. We have hundreds of collections of large binary files (video, audio, archival-quality scans) and their associated textual metadata. We're using Git and git-annex to manage all these files and sync them between locations. Some of the collection repositories are pretty big. One collection has 21000 5MB JPGs. Another has 283 190MB files. Some operations do indeed take time, but you expect that when you're working with so many files. git-annex is awesome!
- jradd 12y ago
  Late response here, but I will have to try this one out as well. git-annex was [virtually] the only result returned from my breif google search. Weird that P4 was no where to be found. Going to have to try both of them out once I get google fiber. :)

montecarl 12y ago

I don't have an answer, but I have an additional complaint. Git itself is pretty terrible at dealing with large repositories if for no other reason than because "git clone" has no resume feature. If your internet connection is interrupted at any point during the initial clone it dies and you have to start over. This means that on an unreliable internet connection it is nearly impossible to clone large repositories.

jerf 12y ago
A git clone is not the same as just rsyncing a .git directory, but if you know Git, the latter can be made to work. You may want to GC the source repo first, and you'll need to manually add a remote and possibly clean up branches after you're done but a quick bash script can finish that up a treat too. (Good task to practice bash scripting on, if you're not familiar with it.)
- bronson 12y ago
  I've done that, waste of time. There's just no excuse for git being unable to resume a clone.
chrismonsanto 12y ago
I've never heard this complaint before. I would have never guessed it would be a problem. Anyway, for the clone, you could just use a different command (such as rsync) to download the other Git repository to your HD, and use git clone on the local path. There is little difference since git clone grabs everything. Subsequent pulls can be done directly to the remote repository.

shoo 12y ago

i've been using unison to sync a checked-out source repository between a host linux machine and a windows vm, over ssh. if you only want uni-directional synchronisation, then maybe just use rsync? this is distinct from your use case as i'm not using it to sync with remote machines, but it might be worth checking out. http://www.cis.upenn.edu/~bcpierce/unison/ http://www.cis.upenn.edu/~bcpierce/unison/

jradd 12y ago
Thanks shoo. I wish this thread wasn't dead so that maybe someone would respond that has used unison. Will check it out if I don't end up loving P4 or git-annex :)

Igglyboo 12y ago

Most game design companies use Perforce specifically for this reason, they have massive asset files that also need to be version controlled and Perforce usually does this better than other choices such as git/hg/svn. Sadly Perforce is proprietary and quite expensive, but it's basically the gold standard in game dev.

forrestthewoods 12y ago

If you need source control and large binary files then use Perforce. It's pretty solved.

jradd 12y ago
Thank you!