3 ms·

I'm surprised nobody has mentioned datalad/git-annex. It might be relevant. It stores references to binary data in a git repository. And depending on the backen

I'm surprised nobody has mentioned datalad/git-annex. It might be relevant. It stores references to binary data in a git repository. And depending on the backend, like bup, it benefit from differential compression. And it is highly peer to peer, and has git-annex-sync[1], which will synchronize branches with its remotes. Datalad is based on git-annex, and already being used for sharing large scientific datasets[2]. [1]: https://git-annex.branchable.com/sync/ https://git-annex.branchable.com/sync/ [2]: https://www.datalad.org/ https://www.datalad.org/

WhatIsDukkha 7y ago
I think git-annex is pretty much the right answer if you are already a linux saavy person that expects a mature battle tested tool that integrates well with your existing linux workflow systems. It sits on top of git in a sensible discernible way. https://www.datalad.org/for/git-users https://www.datalad.org/for/git-users Peoples non text data is going to be some form of indiscernible binary blob to whatever system is managing it. Think about what a diff looks like for a jpeg, excel file, or an audio file. http://docs.datalad.org/en/latest/metadata.html http://docs.datalad.org/en/latest/metadata.html What about git-annex vs git-lfs? 1. """LFS Test Server is an example server that implements the Git LFS API. It is intended to be used for testing the Git LFS client and is not in a production ready state.""" 2. git-lfs seems to want to store your files in the github/microsoft cloud, its not really ready to be deployed inside an existing systems workflow? If you need to check a tickbox and just assume Atlassian/Microsoft has you covered it might be a good choice. 3. git annex has a ton of integrations with multiple backends - http://www.chiark.greenend.org.uk/doc/git-annex/html/special_remotes.html http://www.chiark.greenend.org.uk/doc/git-annex/html/special... 4. Poke around the git-annex website. I think you'll see a tool that's being used in a variety of workflows successfully for a decade plus. That's not true of really any of the other tools others have noted in this thread.
- apostacy 7y ago
  I've worked with both over the last few years. Git LFS is really not p2p like git-annex is. And it is less flexible. But maybe more stable, and git works with it out of the box. You really have to pay for specialized LFS hosting, and you can't remove hosted LFS objects. And while it is possible to only download some LFS objects, it is not easy or intuitive. Once a repo gets to be beyond a certain size, it an be really impractical. Git annex lets you fairly easily get only the objects you want. And it is strongly p2p. You can just set up another repo as a remote, and git-annex-copy whatever files you want to directly over ssh. Or, you can use Google Drive or DropBox or any number of other hosting services that git-annex knows, and stash your files there. Sibling repos will be updated about where they can find the objects. And it is really easy to push your binaries to as many backends as you want. Finally, you can find massive multi terabyte repos of medical image data[1]. There's no way that LFS could handle these terabyte repos. It is really easy to fork one of these data repos, change it, put your changed files into your own backend, and then reshare it. [1]: http://datasets.datalad.org/ http://datasets.datalad.org/
  - WhatIsDukkha 7y ago
    Also of note (from my understanding) is that git lfs local repo is double the size of the underlying data because it doesn't hardlink.
    - deleted 7y ago
      [deleted]

WhatIsDukkha 7y ago

I think git-annex is pretty much the right answer if you are already a linux saavy person that expects a mature battle tested tool that integrates well with your existing linux workflow systems. It sits on top of git in a sensible discernible way. https://www.datalad.org/for/git-users https://www.datalad.org/for/git-users Peoples non text data is going to be some form of indiscernible binary blob to whatever system is managing it. Think about what a diff looks like for a jpeg, excel file, or an audio file. http://docs.datalad.org/en/latest/metadata.html http://docs.datalad.org/en/latest/metadata.html What about git-annex vs git-lfs? 1. """LFS Test Server is an example server that implements the Git LFS API. It is intended to be used for testing the Git LFS client and is not in a production ready state.""" 2. git-lfs seems to want to store your files in the github/microsoft cloud, its not really ready to be deployed inside an existing systems workflow? If you need to check a tickbox and just assume Atlassian/Microsoft has you covered it might be a good choice. 3. git annex has a ton of integrations with multiple backends - http://www.chiark.greenend.org.uk/doc/git-annex/html/special_remotes.html http://www.chiark.greenend.org.uk/doc/git-annex/html/special... 4. Poke around the git-annex website. I think you'll see a tool that's being used in a variety of workflows successfully for a decade plus. That's not true of really any of the other tools others have noted in this thread.

apostacy 7y ago
I've worked with both over the last few years. Git LFS is really not p2p like git-annex is. And it is less flexible. But maybe more stable, and git works with it out of the box. You really have to pay for specialized LFS hosting, and you can't remove hosted LFS objects. And while it is possible to only download some LFS objects, it is not easy or intuitive. Once a repo gets to be beyond a certain size, it an be really impractical. Git annex lets you fairly easily get only the objects you want. And it is strongly p2p. You can just set up another repo as a remote, and git-annex-copy whatever files you want to directly over ssh. Or, you can use Google Drive or DropBox or any number of other hosting services that git-annex knows, and stash your files there. Sibling repos will be updated about where they can find the objects. And it is really easy to push your binaries to as many backends as you want. Finally, you can find massive multi terabyte repos of medical image data[1]. There's no way that LFS could handle these terabyte repos. It is really easy to fork one of these data repos, change it, put your changed files into your own backend, and then reshare it. [1]: http://datasets.datalad.org/ http://datasets.datalad.org/
- WhatIsDukkha 7y ago
  Also of note (from my understanding) is that git lfs local repo is double the size of the underlying data because it doesn't hardlink.
  - deleted 7y ago
    [deleted]