6 ms·

Here's why I'm excited about this: Data Versioning Github is awesome for science. But my code and workflows often are just as reliant on the data I have as the

by thewopr 12y ago

Here's why I'm excited about this: Data Versioning Github is awesome for science. But my code and workflows often are just as reliant on the data I have as the code I've written. With Github, I always have to treat my data separately. A different workflow, a different storage location, different (and often manual) versioning. Now, I can start integrating my data into the same workflow. No size limits. Just drop it in and version it like everything else. 1GB? No problem. 500GB? Just pay the money. This is especially awesome because, as a scientist/dev, I do not want to stand up my own infrastructure and servers/vms. I don't want to do linux updates. I don't want to worry about things going down. It just needs to be there when I need it. When I don't need it anymore, I'll take it down. Done and done.

blaedj 12y ago
I've run into problems with large-ish files in git repos, binaries accidentally committed etc. I'm genuinely curious, is there a good way to use git for the size repositories that you mention?
- bkeroack 12y ago
  Git is great for smaller binaries. Ideal in fact, given that it stores differences between revisions as binary deltas. For large >1GB files, I believe the diffing algorithm is the limiting factor (I would be interested in getting confirmation of that, though). For those files something like git-annex is useful (http://git-annex.branchable.com/ http://git-annex.branchable.com/) I've used git to push around a lot of binary application packages and it's very nice. Previously I was copying around 250-300MB of binaries for every deployment--after switching to a git workflow (via Elita) the binary changesets were typically around 12MB or so.
  - blaedj 12y ago
    Hmm, I'll have to take a look at git-annex, I've run across it but never investigated. Thanks!
- notduncansmith 12y ago
  I don't know of a great solution for Git, but I've heard that Perforce is more suited to handling large binaries - I believe it's used in many game development studios, where binary assets can number in the high gigabytes.
  - Jare 12y ago
    Game developers need both support for large files and for exclusive locking, since assets like images, audio and 3d models are almost never mergeable.
- thewopr 12y ago
  I've had no trouble with non-github git repos handling text files well into the hundreds of megabytes. I haven't pushed the limit on this yet, but for me, personally, my datasets are often many hundreds to thousands of individual files that are medium sized. So it tends to work fairly well. Singular large files may not scale well with git.
rynop 12y ago
Have you ever looked into using an s3 backed git repo via jgit? Not sure how many files/revisions you have but it may be a cost effective alternative that you can start using today. We use it for auto deploys from autoscaled instances cuz github has poor uptime.
- thewopr 12y ago
  No, but I don't understand what this is. Java git? Bound to eclipse? I don't really understand how this would be different than me storing a .git repo locally and replicating it to S3.
  - rynop 12y ago
    jgit is a java implementation of git, that just so happens to have built in s3 support. You can get a self contained cli (jgit.sh) from here: https://eclipse.org/jgit/download/ https://eclipse.org/jgit/download/ . It does not have an eclipse dependency. you can then just add an s3 remote and push to this remote as part of your CI flow. Here is a decent article that describes the end-to-end: http://www.fancybeans.com/blog/2012/08/24/how-to-use-s3-as-a-private-git-repository/ http://www.fancybeans.com/blog/2012/08/24/how-to-use-s3-as-a...
- amalag 12y ago
  I love my jgit s3 git repos. It is super cheap almost free git storage/backup.
m0th87 12y ago
If you're looking for an open source system for managing fat assets in git, we made a tool that integrates with S3: https://github.com/dailymuse/git-fit https://github.com/dailymuse/git-fit
- mdonahoe 12y ago
  How does this compare with git-fat?
  - m0th87 12y ago
    Git-fat uses smudge/clean filters. We used git-media which employs the same technique, and it didn't work well for us. See the section about git-media in the readme.
    - theli0nheart 12y ago
      I maintain a similar tool to manage large files in Git [1], but went the clean/smudge filter route. I think git-media gets into states where it can process files twice even though it shouldn't. From the Git docs: > For best results, clean should not alter its output further if it is run twice ("clean→clean" should be equivalent to "clean"), and multiple smudge commands should not alter clean's output ("smudge→smudge→clean" should be equivalent to "clean"). So this isn't the fault of clean/smudge filters, just the way they were used with git-media. [1]: https://github.com/lionheart/git-bigstore https://github.com/lionheart/git-bigstore
      - m0th87 12y ago
        We experimented with smudge/clean filters with our own implementation and they just didn't seem like the right solution for fat asset management. The most frustrating problem was that filters are executed pretty frequently throughout git workflows, e.g. on `git diff`, even though assets rarely ever change. The added time (though individually small) created a jarring experience. I'd also be curious how git-bigstore addresses conflicts. It seems like a lot of the filter-based tools out there don't handle them well for some reason.
mooreds 12y ago
> I don't want to do linux updates. I'm no EC2 expert, but I believe you still have to do linux updates if you are running on AWS.
- cddotdotslash 12y ago
  If you use Amazon's AMIs they do a lot of updates for you. Recently, they updated their AMI for the bash and SSL security exploits. If you have a permanent instance, you would have to run some updates yourself, but if you use auto-scaling, you can just update the group and then when the instances cycle, the new updates will be applied.
  - Andys 12y ago
    This is also the case with Ubuntu (the most popular Linux distro). The AMIs get updated regularly.
- thewopr 12y ago
  Not for their hosted version management solution, I assume.
AnnaAnisin 12y ago
Domino has a nice solution for version control and reproducibility in the context of analytical / data science work: they keep a snapshot of your code, data, and results every time you run your code. So it’s like version control plus continuous integration. Supports large data files, R, Python, etc. http://www.dominodatalab.com/ http://www.dominodatalab.com/
- thewopr 12y ago
  Yeah, but their system is super simplistic and doesn't support half of the diverse and robust operations that Git and Github do. Can I see version history? Are changes compressed into diffs for text files? etc, etc.
  - merlincorey 12y ago
    Git stores whole objects, not diffs, in case you were confused.
    - yuubi 12y ago
      Git stores diffs between revisions too: see https://www.kernel.org/pub/software/scm/git/docs/git-pack-objects.html https://www.kernel.org/pub/software/scm/git/docs/git-pack-ob... Packs happen automatically sometimes per https://www.kernel.org/pub/software/scm/git/docs/git-gc.html https://www.kernel.org/pub/software/scm/git/docs/git-gc.html
gms 12y ago
What are the disadvantages to using plain Git for your data?
- deleted 12y ago
  [deleted]
  - thewopr 12y ago
    What if your data are formatting in text? Why would branching be any more expensive for data than for code then?
    - deleted 12y ago
      [deleted]
  - xahrepap 12y ago
    I don't think it's branching. From what I understand, it's changing the file. Rather than storing a diff for non-text files, it just keeps the old versions around. So if you store a lot of binary files that change very often, it makes your repo huge. We had this problem before we moved to a dependency manager. Our repo was almost 2gb, even though the checked out code was well under 1gb.
    - deleted 12y ago
      [deleted]
      - xahrepap 12y ago
        The way I understand it, branches in git are essentially just a pointer to a revision, which automatically gets updated as you make changes. Branches don't have any overhead.
- chousuke 12y ago
  Git's storage model is such that every commit is physically a snapshot of the entire work tree. Git makes this efficient with delta compression (essentially, deduplication) which is extremely efficient for text (in fact an entire git repository can be smaller than a single SVN checkout) but is less effective with large binary files since changes don't produce compressible deltas.
xpaulbettsx 12y ago
I would ramp down your excitement, this doesn't solve any of the problems that Git has around large files. Dropping in a 1GB data file into your Git repo will cause problems for you, even if the server doesn't block it like GitHub does.
- twerquie 12y ago
  They mention large binaries on the product page enough times to make me think that they're rewriting Git.
  - xpaulbettsx 12y ago
    Unless that version of Git is also running on your computer, it doesn't change anything. The problem is with local files, not remote ones
  - gfodor 12y ago
    I'm not sure that a server-side rewrite addresses the core issue. Git is designed so that a git fetch pulls the entire repo down. The reason large binary files are a problem is because you are going to pull down the entire history of those files in order to clone the repo. Unless they're also shipping a custom git client (unlikely) then I don't see how they can solve this problem on their end. edit: Come to think of it, the feature they could potentially implement their side is somehow making it so binary files have their history purged somehow. I'm not a git expert but if they are able to trick git into think the files are new, and have no history, each time they change then I guess that could work.
    - cowsandmilk 12y ago
      the core issue is not fetches. The last time I tried to use git to manage scientific data, commits sometimes took ~10 hours. And this was with all text files, not binaries. This was ~100k files and 20 gigs of data.
- thewopr 12y ago
  Oh sure, but at least my limitation in the future will be Git being able to process and handle large files (and maybe the cost of storage) and not Github's file and repo limit.
bonestamp2 12y ago
I came to say the same thing. We have a lot of videos in one of our projects and therefore we can't host it on github due to the size limits.
- elwell 12y ago
  .gitignore isn't a viable option?
  - yoklov 12y ago
    Not if you need them versioned.
theli0nheart 12y ago
We've build a Git extension (git-bigstore) that helps manage large files in Git. It uses a combination of smudge/clean filters and git-notes to store upload/download history, and integrates nicely with S3, Google Cloud Storage, and Rackspace Cloud. Just plop in which filetypes you'd like to not store in your repo in .gitattributes and you're good to go. Our team has been using it for a while now to keep track of large image assets for our web development projects. Cheers! https://github.com/lionheart/git-bigstore https://github.com/lionheart/git-bigstore
joshuak 12y ago
You may not be aware that data versioning is a built in feature of S3[1]. No VMs, no management, just put all stuff in one place and it's versioned automatically. As others have pointed out Git is not a great way to do big data versioning. Git is almost exactly the wrong tool for the job, almost any other revision control system handles large file versioning better because you're not expected to clone entire archive histories to everyone. [1]: https://docs.aws.amazon.com/AmazonS3/latest/UG/enable-bucket-versioning.html https://docs.aws.amazon.com/AmazonS3/latest/UG/enable-bucket...

by thewopr 12y ago

blaedj 12y ago

I've run into problems with large-ish files in git repos, binaries accidentally committed etc. I'm genuinely curious, is there a good way to use git for the size repositories that you mention?

bkeroack 12y ago
Git is great for smaller binaries. Ideal in fact, given that it stores differences between revisions as binary deltas. For large >1GB files, I believe the diffing algorithm is the limiting factor (I would be interested in getting confirmation of that, though). For those files something like git-annex is useful (http://git-annex.branchable.com/ http://git-annex.branchable.com/) I've used git to push around a lot of binary application packages and it's very nice. Previously I was copying around 250-300MB of binaries for every deployment--after switching to a git workflow (via Elita) the binary changesets were typically around 12MB or so.
- blaedj 12y ago
  Hmm, I'll have to take a look at git-annex, I've run across it but never investigated. Thanks!
notduncansmith 12y ago
I don't know of a great solution for Git, but I've heard that Perforce is more suited to handling large binaries - I believe it's used in many game development studios, where binary assets can number in the high gigabytes.
- Jare 12y ago
  Game developers need both support for large files and for exclusive locking, since assets like images, audio and 3d models are almost never mergeable.
thewopr 12y ago
I've had no trouble with non-github git repos handling text files well into the hundreds of megabytes. I haven't pushed the limit on this yet, but for me, personally, my datasets are often many hundreds to thousands of individual files that are medium sized. So it tends to work fairly well. Singular large files may not scale well with git.

rynop 12y ago

Have you ever looked into using an s3 backed git repo via jgit? Not sure how many files/revisions you have but it may be a cost effective alternative that you can start using today. We use it for auto deploys from autoscaled instances cuz github has poor uptime.

thewopr 12y ago
No, but I don't understand what this is. Java git? Bound to eclipse? I don't really understand how this would be different than me storing a .git repo locally and replicating it to S3.
- rynop 12y ago
  jgit is a java implementation of git, that just so happens to have built in s3 support. You can get a self contained cli (jgit.sh) from here: https://eclipse.org/jgit/download/ https://eclipse.org/jgit/download/ . It does not have an eclipse dependency. you can then just add an s3 remote and push to this remote as part of your CI flow. Here is a decent article that describes the end-to-end: http://www.fancybeans.com/blog/2012/08/24/how-to-use-s3-as-a-private-git-repository/ http://www.fancybeans.com/blog/2012/08/24/how-to-use-s3-as-a...
amalag 12y ago
I love my jgit s3 git repos. It is super cheap almost free git storage/backup.

m0th87 12y ago

If you're looking for an open source system for managing fat assets in git, we made a tool that integrates with S3: https://github.com/dailymuse/git-fit https://github.com/dailymuse/git-fit

mdonahoe 12y ago
How does this compare with git-fat?
- m0th87 12y ago
  Git-fat uses smudge/clean filters. We used git-media which employs the same technique, and it didn't work well for us. See the section about git-media in the readme.
  - theli0nheart 12y ago
    I maintain a similar tool to manage large files in Git [1], but went the clean/smudge filter route. I think git-media gets into states where it can process files twice even though it shouldn't. From the Git docs: > For best results, clean should not alter its output further if it is run twice ("clean→clean" should be equivalent to "clean"), and multiple smudge commands should not alter clean's output ("smudge→smudge→clean" should be equivalent to "clean"). So this isn't the fault of clean/smudge filters, just the way they were used with git-media. [1]: https://github.com/lionheart/git-bigstore https://github.com/lionheart/git-bigstore
    - m0th87 12y ago
      We experimented with smudge/clean filters with our own implementation and they just didn't seem like the right solution for fat asset management. The most frustrating problem was that filters are executed pretty frequently throughout git workflows, e.g. on `git diff`, even though assets rarely ever change. The added time (though individually small) created a jarring experience. I'd also be curious how git-bigstore addresses conflicts. It seems like a lot of the filter-based tools out there don't handle them well for some reason.

mooreds 12y ago

> I don't want to do linux updates. I'm no EC2 expert, but I believe you still have to do linux updates if you are running on AWS.

cddotdotslash 12y ago
If you use Amazon's AMIs they do a lot of updates for you. Recently, they updated their AMI for the bash and SSL security exploits. If you have a permanent instance, you would have to run some updates yourself, but if you use auto-scaling, you can just update the group and then when the instances cycle, the new updates will be applied.
- Andys 12y ago
  This is also the case with Ubuntu (the most popular Linux distro). The AMIs get updated regularly.
thewopr 12y ago
Not for their hosted version management solution, I assume.

AnnaAnisin 12y ago

Domino has a nice solution for version control and reproducibility in the context of analytical / data science work: they keep a snapshot of your code, data, and results every time you run your code. So it’s like version control plus continuous integration. Supports large data files, R, Python, etc. http://www.dominodatalab.com/ http://www.dominodatalab.com/

thewopr 12y ago
Yeah, but their system is super simplistic and doesn't support half of the diverse and robust operations that Git and Github do. Can I see version history? Are changes compressed into diffs for text files? etc, etc.
- merlincorey 12y ago
  Git stores whole objects, not diffs, in case you were confused.
  - yuubi 12y ago
    Git stores diffs between revisions too: see https://www.kernel.org/pub/software/scm/git/docs/git-pack-objects.html https://www.kernel.org/pub/software/scm/git/docs/git-pack-ob... Packs happen automatically sometimes per https://www.kernel.org/pub/software/scm/git/docs/git-gc.html https://www.kernel.org/pub/software/scm/git/docs/git-gc.html

gms 12y ago

What are the disadvantages to using plain Git for your data?

deleted 12y ago
[deleted]
- thewopr 12y ago
  What if your data are formatting in text? Why would branching be any more expensive for data than for code then?
  - deleted 12y ago
    [deleted]
- xahrepap 12y ago
  I don't think it's branching. From what I understand, it's changing the file. Rather than storing a diff for non-text files, it just keeps the old versions around. So if you store a lot of binary files that change very often, it makes your repo huge. We had this problem before we moved to a dependency manager. Our repo was almost 2gb, even though the checked out code was well under 1gb.
  - deleted 12y ago
    [deleted]
    - xahrepap 12y ago
      The way I understand it, branches in git are essentially just a pointer to a revision, which automatically gets updated as you make changes. Branches don't have any overhead.
chousuke 12y ago
Git's storage model is such that every commit is physically a snapshot of the entire work tree. Git makes this efficient with delta compression (essentially, deduplication) which is extremely efficient for text (in fact an entire git repository can be smaller than a single SVN checkout) but is less effective with large binary files since changes don't produce compressible deltas.

xpaulbettsx 12y ago

I would ramp down your excitement, this doesn't solve any of the problems that Git has around large files. Dropping in a 1GB data file into your Git repo will cause problems for you, even if the server doesn't block it like GitHub does.

twerquie 12y ago
They mention large binaries on the product page enough times to make me think that they're rewriting Git.
- xpaulbettsx 12y ago
  Unless that version of Git is also running on your computer, it doesn't change anything. The problem is with local files, not remote ones
- gfodor 12y ago
  I'm not sure that a server-side rewrite addresses the core issue. Git is designed so that a git fetch pulls the entire repo down. The reason large binary files are a problem is because you are going to pull down the entire history of those files in order to clone the repo. Unless they're also shipping a custom git client (unlikely) then I don't see how they can solve this problem on their end. edit: Come to think of it, the feature they could potentially implement their side is somehow making it so binary files have their history purged somehow. I'm not a git expert but if they are able to trick git into think the files are new, and have no history, each time they change then I guess that could work.
  - cowsandmilk 12y ago
    the core issue is not fetches. The last time I tried to use git to manage scientific data, commits sometimes took ~10 hours. And this was with all text files, not binaries. This was ~100k files and 20 gigs of data.
thewopr 12y ago
Oh sure, but at least my limitation in the future will be Git being able to process and handle large files (and maybe the cost of storage) and not Github's file and repo limit.

bonestamp2 12y ago

I came to say the same thing. We have a lot of videos in one of our projects and therefore we can't host it on github due to the size limits.

elwell 12y ago
.gitignore isn't a viable option?
- yoklov 12y ago
  Not if you need them versioned.

theli0nheart 12y ago

We've build a Git extension (git-bigstore) that helps manage large files in Git. It uses a combination of smudge/clean filters and git-notes to store upload/download history, and integrates nicely with S3, Google Cloud Storage, and Rackspace Cloud. Just plop in which filetypes you'd like to not store in your repo in .gitattributes and you're good to go. Our team has been using it for a while now to keep track of large image assets for our web development projects. Cheers! https://github.com/lionheart/git-bigstore https://github.com/lionheart/git-bigstore

joshuak 12y ago

You may not be aware that data versioning is a built in feature of S3[1]. No VMs, no management, just put all stuff in one place and it's versioned automatically. As others have pointed out Git is not a great way to do big data versioning. Git is almost exactly the wrong tool for the job, almost any other revision control system handles large file versioning better because you're not expected to clone entire archive histories to everyone. [1]: https://docs.aws.amazon.com/AmazonS3/latest/UG/enable-bucket-versioning.html https://docs.aws.amazon.com/AmazonS3/latest/UG/enable-bucket...