5 ms·

Has someone had a closer look and can say how this compares to Git-Annex?

by Rondom 12y ago

jefurii 12y ago
This and git-annex (and git-fat and others) use the same basic architecture of storing links in Git and schlepping the binaries around separately. Git-annex renames binaries with their SHA256 hashes, puts them in a .git/annex/ dir, and replaces files in the working dir with symlinks. Git-LFS seems to use small metadata pointer files (SHA256 hash, file size, git-lfs version) instead of symlinks. Not sure whether the files reside in something like the .git/annex/ dir; I'm guessing the do or there wouldn't be those pointer files. You can clone a repo without having to download the files. With git-annex you can sync between non-bare and bare repositories without having a central server. Git-LFS seems to have a separate server for binaries. It looks like it may act like a git-annex special remote rather than git-annex's usage of synced/master branch. Git-annex repos share information about the locations of annex files, how many repos contain a given file, etc. You can trust and un-trust repos. It doesn't look like Git-LFS offers this. Git-LFS has a REST API. I'm using an old version of git-annex so I can't say if it does (I think it does). Git-LFS is written in Go, git-annex is Haskell. Git-LFS is a GitHub project. GitHub will offer object hosting. Update: clarity, speling, added a bullet point.
- icebraining 12y ago
  git annex uses symlinks in indirect mode, but can use small files in direct mode (useful for filesystems which don't support symlinks, like many Android sdcards).
- joeyh 12y ago
  The lack of location tracking looks like the most significant difference to me. While the git-lfs documentation does mention that different git remotes can have different LFS endpoints configured, all git-lfs knows about a file is its SHA256. So how can it tell which remote to download the file from? The best it could do is try different remotes until it finds one that has the file. I hesitate to say this means git-lfs is not distributed at all, but it seems significantly less distributed than git-annex, which can keep track of files that might be in Glacier, or on an offline drive, or a repo cloned on a nearby computer, and so can be used in a more peer-to-peer fashion when storing and retrieving the large files.
  - deleted 12y ago
    [deleted]
  - paulcody 12y ago
    At a minimum, the metafile should include a URI, stating the last known location of the file, rather than just a bare SHA256.
    - joeyh 12y ago
      That doesn't work very well. Consider what happens if two different clones of a repo update the metadata last-known url at the same time with different urls. Merge conflicts. This is why git-annex uses a separate branch for location tracking information, which it can merge in a conflict-free manner.
      - paulcody 12y ago
        Late reply: that's fine, as long as there is some information as to where the location is, and not GitHub-only.
ansible 12y ago
I think it is much closer to git-fat [1] (as jefurii mentions). I've been trying out git-fat on a large repository that has some binaries in it. Since we are already using ssh to access the server (and have the usual ssh-agent setup), it was easy to integrate it with scp. Unfortunately, the performance for our workload (45K files, 5GB total) with git-fat wasn't much different than plain git. It seems most of my problems stem from the number of files, rather than the size. If I had a smaller number of very large files, git-fat might be a good solution. [1] https://github.com/jedbrown/git-fat https://github.com/jedbrown/git-fat

Has someone had a closer look and can say how this compares to Git-Annex?

by Rondom 12y ago

Has someone had a closer look and can say how this compares to Git-Annex?

This and git-annex (and git-fat and others) use the same basic architecture of storing links in Git and schlepping the binaries around separately. Git-annex renames binaries with their SHA256 hashes, puts them in a .git/annex/ dir, and replaces files in the working dir with symlinks. Git-LFS seems to use small metadata pointer files (SHA256 hash, file size, git-lfs version) instead of symlinks. Not sure whether the files reside in something like the .git/annex/ dir; I'm guessing the do or there wouldn't be those pointer files. You can clone a repo without having to download the files. With git-annex you can sync between non-bare and bare repositories without having a central server. Git-LFS seems to have a separate server for binaries. It looks like it may act like a git-annex special remote rather than git-annex's usage of synced/master branch. Git-annex repos share information about the locations of annex files, how many repos contain a given file, etc. You can trust and un-trust repos. It doesn't look like Git-LFS offers this. Git-LFS has a REST API. I'm using an old version of git-annex so I can't say if it does (I think it does). Git-LFS is written in Go, git-annex is Haskell. Git-LFS is a GitHub project. GitHub will offer object hosting. Update: clarity, speling, added a bullet point.

icebraining 12y ago
git annex uses symlinks in indirect mode, but can use small files in direct mode (useful for filesystems which don't support symlinks, like many Android sdcards).
joeyh 12y ago
The lack of location tracking looks like the most significant difference to me. While the git-lfs documentation does mention that different git remotes can have different LFS endpoints configured, all git-lfs knows about a file is its SHA256. So how can it tell which remote to download the file from? The best it could do is try different remotes until it finds one that has the file. I hesitate to say this means git-lfs is not distributed at all, but it seems significantly less distributed than git-annex, which can keep track of files that might be in Glacier, or on an offline drive, or a repo cloned on a nearby computer, and so can be used in a more peer-to-peer fashion when storing and retrieving the large files.
- deleted 12y ago
  [deleted]
- paulcody 12y ago
  At a minimum, the metafile should include a URI, stating the last known location of the file, rather than just a bare SHA256.
  - joeyh 12y ago
    That doesn't work very well. Consider what happens if two different clones of a repo update the metadata last-known url at the same time with different urls. Merge conflicts. This is why git-annex uses a separate branch for location tracking information, which it can merge in a conflict-free manner.
    - paulcody 12y ago
      Late reply: that's fine, as long as there is some information as to where the location is, and not GitHub-only.

ansible 12y ago

I think it is much closer to git-fat [1] (as jefurii mentions). I've been trying out git-fat on a large repository that has some binaries in it. Since we are already using ssh to access the server (and have the usual ssh-agent setup), it was easy to integrate it with scp. Unfortunately, the performance for our workload (45K files, 5GB total) with git-fat wasn't much different than plain git. It seems most of my problems stem from the number of files, rather than the size. If I had a smaller number of very large files, git-fat might be a good solution. [1] https://github.com/jedbrown/git-fat https://github.com/jedbrown/git-fat