10 ms·
Git Large File Storage 1.0
- et1337 11y agoI haven't been following the various Git large file solutions - can someone comment on how this implementation compares to git-annex or whatever else is out there?
- Amorymeltzer 11y agoWe had a pretty good discussion here when this was initially announced six months ago: https://news.ycombinator.com/item?id=9343021 https://news.ycombinator.com/item?id=9343021 Looks like they've had some success!
- rch 11y agoI wonder if it could leverage the HDF5 diff tool somehow... https://www.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Diff https://www.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Diff And perhaps GridFTP: http://toolkit.globus.org/toolkit/docs/latest-stable/gridftp/ http://toolkit.globus.org/toolkit/docs/latest-stable/gridftp...
- m0th87 11y agoIf for whatever reason lfs doesn't work for you, check out our solution to large file storage on git: https://github.com/dailymuse/git-fit https://github.com/dailymuse/git-fit We wrote it because of various issues with tools at the time, basically boiling down to an inclination to have a dead simple solution. I haven't tried lfs, but if it's anything like github's other software, then I'm sure it's substantially better than our tool.
- sytse 11y agoThis is great, we plan to ship alpha LFS support in GitLab CE & EE & .com in 8.1 or 8.2. That is in addition to the git annex support that EE & .com already have for a longer time https://about.gitlab.com/2015/02/17/gitlab-annex-solves-the-problem-of-versioning-large-binaries-with-git/ https://about.gitlab.com/2015/02/17/gitlab-annex-solves-the-...
- zertrin 11y agoIs it safe to assume that Gitlab's implementation of Git LFS will allow to host the file storage server on premises and potentially on another machine than the one running Gitlab?
- sytse 11y agoIt for sure will allow you to host it on premises. We'll open source our LFS server so people can use it for other purposes.
- nixgeek 11y agoWill you be supporting both indefinitely or is there a plan to transition to a single well-supported solution for large files over the coming N releases?
- sytse 11y agoIt seems that LFS is winning over Annex here, we might drop Annex in 9.x, but nothing is certain.
- porker 11y agoIf LFS is winning but Annex is the better solution (as the comments I've read point to), it would be a shame to drop Annex.
- sytse 11y agoThere is a cost to everything and we try to be pragmatic. Git Annex is causing lots of work for us in the Omnibus packages. Video2000 is also technically better than VHS, but still people stopped producing equipment for it much sooner.
- onionjake 11y agoWhile I'm sure this will help some people use git to address a use case that was previously impossible with git, I can't help but feel that it a bad step overall for the git ecosystem. It appears to centralize a distributed version control with no option to continue to use it in a distributed fashion. What would be wrong with fixing/enhancing the existing git protocols to enable shallow fetching of a commit (I want commit A, but without objects B and C, which are huge). Git already fully supports working from a shallow clone (not the full history) so it wouldn't be too much of a stretch to make it work with shallow trees (I didn't fetch all of the objects). I'm sure git LFS was the quickest way for github to support a use case, but I'm not sure it is the best thing for git.
- sytse 11y agoI don't see why the centralization is different, can't you just download all large files and upload them somewhere else?
- onionjake 11y agoHow? Does git-lfs have a way of downloading and serving all of the files? If it was fully integrated with git, you could do 'git fetch --all-objects-ever' and your repository could then be cloned and fetched from.
- sytse 11y ago`git lfs fetch -all` should download everything https://github.com/github/git-lfs/blob/master/docs/man/git-lfs-fetch.1.ronn#L27 https://github.com/github/git-lfs/blob/master/docs/man/git-l...
- onionjake 11y agoNeat! Is there a way for me to serve those files so people can then use my repository as the authoritative source? I see that they still have the concepts of remotes, so maybe things are getting there?
- k__ 11y agoIs there a solution, that doesn't depend on external storage? I have data that belongs to my source, but is rather big and I want it inside of my repo.
- m12k 11y agoThey do have a reference implementation of the serverside here: https://github.com/github/lfs-test-server https://github.com/github/lfs-test-server - though they themselves don't consider it production ready. But I'm sure it'll either get there in time, or another open source implementation will rise to the challenge (cf. syste's comment about GitLab planning support for this: https://news.ycombinator.com/item?id=10313495 https://news.ycombinator.com/item?id=10313495 )
- jonesetc 11y agoAny information on GitHub Enterprise support?
- rlegit 11y agoGitHub Enterprise has been supporting LFS since 2.2 (current latest is 2.3.3) in a technical preview mode. See here: https://enterprise.github.com/releases/2.2.0/notes https://enterprise.github.com/releases/2.2.0/notes
- res0nat0r 11y agoWhere are the files actually stored? I hear "git lfs server" in the demo video, can this be changed? Can I init my repo and tell it to push all my objects to my own private s3 bucket, or can I only rely on some outside lfs server I don't control?
- thristian 11y agoNot with GitLFS, since it's designed under the assumption that the hostname serving your repo over ssh is also the GitLFS server over HTTP. However git-fat[1] is an alternative system that works in much the same way, but lets you configure where the files are stored. [1]: https://github.com/cyaninc/git-fat https://github.com/cyaninc/git-fat
- chejazi 11y agoAs someone new to this idea, the README helped clarify the workflow: https://github.com/github/git-lfs/blob/master/docs/api/README.md https://github.com/github/git-lfs/blob/master/docs/api/READM...
- vvanders 11y agoWithout locking this is largely useless. Usually large files are binary blobs(PSD, .ma, etc) and it becomes incredibly easy to blow away someone's work by not pulling before every file you edit(or two people edit at the same time). As much as some people hate Perforce that's exactly what they are setup to do. Plus their binary syncing algorithms are top-notch. We used to regularly pull ~300GB art repo(for gamedev) in ~20 minutes. Git is great for code but this seems like square peg, round hole to me.
- alkonaut 11y agoWon't a central storage for the large files alao make it straightforward to add locking functionality in a future version of git-lfs, or as an add-in? I agree it sure looks like an omission to have a vcs that is centralized and aimed at binary data, without having any locking functionality.
- vvanders 11y agoYeah, but that's assuming a whole lot of infrastructure around marking files read-only. I also haven't seen what the pull performance is like, P4 is a pretty known quantity(with caching as well).
- alkonaut 11y agoI don't even necessarily need my client to mark files as read only on my local machine. A system that just lets me query who has the lock on a certain file, and lets me get the lock on it if noone has, is miles ahead of shouting/email. A remote-only locking system should be pretty easy to implement, e.g by just throwing in "filename.userid.lock" into the filesystem next to the file in question.
- sytse 11y agoWe plan to add locking to the GitLab web ui at some point. At least this will prevent two people from locking the same file.
- 11y ago
- m12k 11y agoI worked at Unity for a couple years and they are one of the biggest users of (and maintainers of) the Mercurial LargeFiles extension, so I was using that on a daily basis. I agree that it should be a measure of last resort, but if you can't avoid working with big binary files, it makes the difference between a workflow that is a bit more cumbersome, and one that just grinds to a halt. Getting this functionality in git is great. And it'll mean a huge step forward in collaboration tools for game developers. You pretty much can't avoid big binary files when making games - and so far they've been stuck with SVN or Perforce (or the more adventurous ones are trying out PlasticSCM, which apparently is pretty nice too, but is proprietary and doesn't have a big ecosystem around it like git does). I hope this can lead to a boom of game developers using git.
- babuskov 11y agoYup. I'm using git for game source code and I'm often holding off any commits to graphics/music until project is done. Any workaround outside git means you have two systems to manage and it can get really painful.
- lspears 11y agoThat video is hilarious! I wish we had more awesome videos like this for new technologies!
- wscott 11y agoThe complaints about git-lfs make me think we need to tell people about BAM for BitKeeper. http://www.bitkeeper.com/features_binary_asset_management_bam http://www.bitkeeper.com/features_binary_asset_management_ba... BAM works with a similar idea, instead of saving large files in the local repository, users are allowed to save them in a centralized server. This saves disk space and network transfer time. However, unlike other solutions, BAM preserves the semantics of distributed development. Instead of requiring a single or standardized set of servers, every user can have a different BAM server. Data is moved between servers automatically and on demand. One group in an office might use a single BAM server for storing all their data close and locally. When another development group is started in India, they can use a server local to them. The binary assets will automatically transfer to the India server as commits are pulled between sites. This allows centralized storage of your data and yet still supports having a team work while completely disconnected from the internet.
- luckydude 11y agoI've been using BAM for quite a while (I'm one of the developers of it). I use it to store my photos. I've 55GB of photos in there and backing them up is cd photos bk push Works pretty well, when my mom was still alive we pushed them to her imac and the screen saver was pointed at that directory. So she got to see the kids and I got another backup.
- entitycontext 11y agoNot sure if they were working together with GitHub on this, but Microsoft also announced today that Visual Studio Online Git repos now support Git-LFS with unlimited free storage: http://blogs.msdn.com/b/visualstudioalm/archive/2015/10/01/announcing-git-lfs-on-all-vso-git-repos.aspx http://blogs.msdn.com/b/visualstudioalm/archive/2015/10/01/a...
- tyoverby 11y agoWhat happens if someone that hasn't downloaded their command line tools tries to clone your repo? Will they get the big files too?
- anotherevan 11y agoWas I the only one who expected that bear to move on its own?
- kazinator 11y agoThis could be a corollary to P. Graham's "don't do anything that scales": don't do anything that involves plonking stupidly large files into version control.
- mindprince 11y agoGood thing that Atlassian/BitBucket would also be supporting it: https://blog.bitbucket.org/2015/10/01/contributing-to-git-lfs/ https://blog.bitbucket.org/2015/10/01/contributing-to-git-lf... And very glad to read that they decided to contribute to this instead of working on their own solution for the same problem. Kudos!
- kannonboy 11y agoThe fact that both Atlassian and GitHub intended to unveil their own almost identical competing solutions, both built in Go, in consecutive sessions at the Git Merge conference (without either being aware of the other) is pretty hilarious.
- elcritch 11y agoGit-lfs has been helpful for managing my repo of scientific research data. Hundreds of large-ish excel files, pngs, and hdf5 add up quickly if you're doing lots of small edits. There's still some warts (don't forget git lfs init after cloning!), buts it's mostly fast and transparent. I also ponied up $5 a month to get 50 gigs or so of lfs storage. Decent deal imho.