8 ms·

This looks really interesting. You basically trade the ability to have diffs (nearly meaningless on binary files anyway) for representing large files as their S

by jxf 12y ago

This looks really interesting. You basically trade the ability to have diffs (nearly meaningless on binary files anyway) for representing large files as their SHA-256 equivalent values on a remote server. What will be interesting is to see whether GitHub's implementation of LFS allows a "bring your own server" option. Right now the answer seems to be no -- the server knows about all the SHAs, and GitHub's server only supports their own storage endpoint. So you couldn't use, say, S3 to host your Git LFS files.

icebraining 12y ago
You basically trade the ability to have diffs (nearly meaningless on binary files anyway) for representing large files as their SHA-256 equivalent values on a remote server. That's exactly what git-annex does. Except it can host on your own servers, or S3, or Tahoe-LAFS, or rsync.net, etc. And it's free software. And it supports multiple servers for the same repo, so you have redundancy. Adding an S3 remote is just setting the AWS keys and running a single command: http://git-annex.branchable.com/tips/using_Amazon_S3/ http://git-annex.branchable.com/tips/using_Amazon_S3/
- azernik 12y ago
  This is also free software, and you can also use your own server.
  - efuquen 12y ago
    I think the question is why did they role their own solution when there was one already an open and freely one available. If it wasn't suitable in some way I would really like to know why.
    - s73v3r 12y ago
      If git-annex really worked well, and was easy to use, I imagine there'd be much more uptake of it.
      - icebraining 12y ago
        "Build it and they will come" Besides, I'm sure Joey Hess wouldn't refuse the help.
        sytse 12y ago
        Joey has been most helpful to us at GitLab, I'm sure he would be happy to help out GitHub too.
- rakoo 12y ago
  And if you want the simplest solution (ie store blobs with the rest of your code), gitlab offers git-annex compatibility (https://about.gitlab.com/2015/02/17/gitlab-annex-solves-the-problem-of-versioning-large-binaries-with-git/ https://about.gitlab.com/2015/02/17/gitlab-annex-solves-the-...)
  - sytse 12y ago
    Thanks for mentioning us rakoo. Using git-annex on GitLab.com is completely free. We're thinking about a 5Gb per repo cap but right now its unlimited.
- sytse 12y ago
  Indeed similar to git-annex, why does GitHub not support that? With GitLab we added support for that recently and it was pretty easy to do. Anyway, interesting that linking to sha's is becoming the default solution. Looking forward to playing with this and comparing the solutions.
  - FireBeyond 12y ago
    Because they are going to charge for LFS, I suspect. Just as they don't support private repos for free, either.
    - sytse 12y ago
      They could also charge for git-annex I think. It is not that they need their own type of private repo's to charge for that. By the way, GitLab.com offers unlimited repo's and 5GB per repo for free.
      - thoughtflretort 12y ago
        How many users/how long will you be able to provide that for free?
        sytse 12y ago
        Forever, our business model is to make money with on-premises software (GitLab EE). In the long term we might add a marketplace to GitLab.com were you can subscribe to additional services (like Heroku does).
cbhl 12y ago
If the extension is open-source, you could probably build your own server implementation and fork the extension to change what server it contacts.
- thedaniel 12y ago
  yes: https://github.com/github/git-lfs/blob/master/docs/api.md https://github.com/github/git-lfs/blob/master/docs/api.md
deleted 12y ago
[deleted]
- deleted 12y ago
  [deleted]
Oompa 12y ago
There's an example server that was also open sourced today: https://github.com/github/lfs-test-server https://github.com/github/lfs-test-server
jordigh 12y ago
> You basically trade the ability to have diffs (nearly meaningless on binary files anyway) for representing large files as their SHA-256 equivalent values on a remote server. That's exactly how Mercurial's largefiles works too: http://mercurial.selenic.com/wiki/LargefilesExtension#The_local_store http://mercurial.selenic.com/wiki/LargefilesExtension#The_lo... Also, you don't need any kind of special server. Any hg repo can turn into a largefiles store by just flipping the bit in the repo configuration.
lebek 12y ago
Nitpick: they gave four examples of large binary files: audio samples, datasets, graphics, and videos. Diffs would be meaningful in every case.
- sjwright 12y ago
  Surely not diffs offered by the git ecosystem?

by jxf 12y ago

icebraining 12y ago

You basically trade the ability to have diffs (nearly meaningless on binary files anyway) for representing large files as their SHA-256 equivalent values on a remote server. That's exactly what git-annex does. Except it can host on your own servers, or S3, or Tahoe-LAFS, or rsync.net, etc. And it's free software. And it supports multiple servers for the same repo, so you have redundancy. Adding an S3 remote is just setting the AWS keys and running a single command: http://git-annex.branchable.com/tips/using_Amazon_S3/ http://git-annex.branchable.com/tips/using_Amazon_S3/

azernik 12y ago
This is also free software, and you can also use your own server.
- efuquen 12y ago
  I think the question is why did they role their own solution when there was one already an open and freely one available. If it wasn't suitable in some way I would really like to know why.
  - s73v3r 12y ago
    If git-annex really worked well, and was easy to use, I imagine there'd be much more uptake of it.
    - icebraining 12y ago
      "Build it and they will come" Besides, I'm sure Joey Hess wouldn't refuse the help.
      - sytse 12y ago
        Joey has been most helpful to us at GitLab, I'm sure he would be happy to help out GitHub too.
rakoo 12y ago
And if you want the simplest solution (ie store blobs with the rest of your code), gitlab offers git-annex compatibility (https://about.gitlab.com/2015/02/17/gitlab-annex-solves-the-problem-of-versioning-large-binaries-with-git/ https://about.gitlab.com/2015/02/17/gitlab-annex-solves-the-...)
- sytse 12y ago
  Thanks for mentioning us rakoo. Using git-annex on GitLab.com is completely free. We're thinking about a 5Gb per repo cap but right now its unlimited.
sytse 12y ago
Indeed similar to git-annex, why does GitHub not support that? With GitLab we added support for that recently and it was pretty easy to do. Anyway, interesting that linking to sha's is becoming the default solution. Looking forward to playing with this and comparing the solutions.
- FireBeyond 12y ago
  Because they are going to charge for LFS, I suspect. Just as they don't support private repos for free, either.
  - sytse 12y ago
    They could also charge for git-annex I think. It is not that they need their own type of private repo's to charge for that. By the way, GitLab.com offers unlimited repo's and 5GB per repo for free.
    - thoughtflretort 12y ago
      How many users/how long will you be able to provide that for free?
      - sytse 12y ago
        Forever, our business model is to make money with on-premises software (GitLab EE). In the long term we might add a marketplace to GitLab.com were you can subscribe to additional services (like Heroku does).

cbhl 12y ago

If the extension is open-source, you could probably build your own server implementation and fork the extension to change what server it contacts.

thedaniel 12y ago
yes: https://github.com/github/git-lfs/blob/master/docs/api.md https://github.com/github/git-lfs/blob/master/docs/api.md

deleted 12y ago

[deleted]

deleted 12y ago
[deleted]

Oompa 12y ago

There's an example server that was also open sourced today: https://github.com/github/lfs-test-server https://github.com/github/lfs-test-server

jordigh 12y ago

> You basically trade the ability to have diffs (nearly meaningless on binary files anyway) for representing large files as their SHA-256 equivalent values on a remote server. That's exactly how Mercurial's largefiles works too: http://mercurial.selenic.com/wiki/LargefilesExtension#The_local_store http://mercurial.selenic.com/wiki/LargefilesExtension#The_lo... Also, you don't need any kind of special server. Any hg repo can turn into a largefiles store by just flipping the bit in the repo configuration.

lebek 12y ago

Nitpick: they gave four examples of large binary files: audio samples, datasets, graphics, and videos. Diffs would be meaningful in every case.

sjwright 12y ago
Surely not diffs offered by the git ecosystem?