6 ms·

Git-Annex

225 points by keepamovin 1y ago

EmilStenstrom 1y ago
Happy to see use cases front and center in command line documentation. They seem to always start with ”obscure command flag that you’ll probably never use”.
ygritte 1y ago
Could this be abused to simulate something like SVN externals? I always found git submodules to be a very bad replacement for that.
fragmede 1y ago
GitHub really embraced the Microsoft-esque NIH with LFS, instead of adopting git-annex.
- keepamovin 1y ago
  To its absolute detriment Here is a talk by a person who adores it: Yann Büchau: Staying in Control of your Scientific Data with Git Annex https://www.youtube.com/watch?v=IdRUsn-zB2s https://www.youtube.com/watch?v=IdRUsn-zB2s
  - codemac 1y ago
    While Yann has built many things with git-annex, we should be clear that the creator of git-annex is relatively singular, Joey Hess.
    - keepamovin 1y ago
      Here is a comment about Joey: https://news.ycombinator.com/item?id=14908529 https://news.ycombinator.com/item?id=14908529 And an interview When power is low, I often hack in the evenings by lantern light. https://usesthis.com/interviews/joey.hess/ https://usesthis.com/interviews/joey.hess/
- mathstuf 1y ago
  While I also find git-annex more elegant, its cross-platform story is weaker. Note that LFS was originally a collaboration between GitHub and Bitbucket (maybe? Some forge vendor I think). One had the implementation and the other had the name. They met at a Git conference and we have what we have today. My main gripes these days are the woefully inadequate limits GitHub has in place for larger projects. Coupled with the "must have all objects locally to satisfy an arbitrary push", any decently sized developer community will blow the limit fairly quickly. FD: I have contributed to git-lfs.
deleted 1y ago
[deleted]
andrewmcwatters 1y ago
git-annex has some really awkward documentation. You can apparently do, sort of, but not really, the same thing git-fetch-file[1] does, with git-annex: git fetch-file add https://github.com/icculus/physfs.git "**" lib/physfs-main git fetch-file pull `add` creates this at `.git-remote-files`: [file "**"] commit = 9d18d36b5a5207b72f473f05e1b2834e347d8144 target = lib/physfs-main repository = https://github.com/icculus/physfs branch = main But git-annex's documentation goes on and on about a bunch of commands I don't really want to read about, whereas those two lines and that .git-remote-files manifest just told you what git-fetch-file does. [1]: https://github.com/andrewmcwattersandco/git-fetch-file https://github.com/andrewmcwattersandco/git-fetch-file
- nolist_policy 1y ago
  Not at all. git-annex is for managing large files in git and unlike git-lfs it preserves the distributed nature of git.
- keepamovin 1y ago
  Here is a guide you might like: https://www.youtube.com/watch?v=p0eVyhv2rbk https://www.youtube.com/watch?v=p0eVyhv2rbk
nolist_policy 1y ago
I use git-annex to manage all my data on all my drives. It automatically keeps track of which files are on which drives, it ensures that there are enough copies and it checksums everything. It works perfectly with offline drives. git-annex can be a bit hard to grasp, so I suggest to create a throw-away repository, following the walkthrough[1] and try things out. See also workflows[2]. [1] https://git-annex.branchable.com/walkthrough/ https://git-annex.branchable.com/walkthrough/ [2] https://git-annex.branchable.com/workflow/ https://git-annex.branchable.com/workflow/
- albertzeyer 1y ago
  How much data do you have? I'm using git-annex on my photos, and that are around 100k-1M files, several TB of data, on a ZFS. In the beginning, everything was fine, but it starts to become increasingly slow, such that every operation takes several minutes (5-30 mins or so). I wonder a bit whether that is ZFS, or git-annex, or maybe my disk, or sth else.
  - riedel 1y ago
    It would be great to have comprehensive benchmarks for git lsf, git annex, dvc and alike. I am also always getting annoyed with one or the other , e.g. due to the hashing overhead, etc. However, in many cases the annoyances come with bad filesystem integration on Windows in my case.
    - rurban 1y ago
      My guess is the windows virus scaner
  - warp 1y ago
    My experience is the same, git-annex just doesn't work well with lots of small files. With annexes on slow USB disks, connected to a Raspberry Pi 3 or 4, I'm already annoyed when working with my largest annex (in file count) of 25000 files. However, I mostly use annex as a way to archive stuff and make sure I have enough copies in distinct physical locations. So for photos I now just tar them up with one .tar file per family member per year. This works fine for for me for any data I want to keep safe but don't need to access directly very often.
Munksgaard 1y ago
Git-Annex is a cool piece of technology, but my impression is that it works best for single-user repositories. So for instance, as @nolist_policy described in a sibling comment, managing all your personal files, documents, music, etc. across many different devices. I tried using it for syncing large files in a collaborative repository, and the use of "magic" branches didn't seem to scale well.
- geephroh 1y ago
  YMMV, but it works for my org. We're an archival institution and have been using git-annex for more than a decade as the storage backend for a digital repository system designed for long-term, robust preservation. Admittedly, we only have 15-20 staff; but +30TB of data, ~750K files (binaries + metadata), across hundreds of collection repos.
  - eigengrau 1y ago
    How (if you saw that need) did you address permissions concerns, e.g., around any Git users being able to force drop all files from a backend? Back (long time ago) when I was looking into this, there was no KISS, out-of-the-box way to manage the Git Annex operations a Git user would be allowed to perform. Gitolite (or whatever Git platform of choice) can address access control concerns for regular Git pushes, but there is no way to define policies on Git Annex operations (configuration, storage management). Might not be super hard to create a Gitolite plugin to address these, but ultimately for my use-case it wasn’t worth the effort (I didn’t really need shared Git Annex repos). Do you tackle these concerns somehow? I guess if people don’t interact with your repositories via Git/SSH but only through some custom UI, you might deal with it there.
ttiurani 1y ago
Relevant discussion 9 days ago about the new native git large object promisers in "The future of large files in Git is Git": https://news.ycombinator.com/item?id=44916783 https://news.ycombinator.com/item?id=44916783
- avar 1y ago
  Thanks, also not-so-relevant, for the reasons I noted in a comment in that thread: https://news.ycombinator.com/item?id=44922405 https://news.ycombinator.com/item?id=44922405 I.e. annex is really in a different problem space than "big files in git", despite the obvious overlap. A good way to think about it is that git-annex is sort of a git-native and distributed solution to the storage problem at the "other side" ("server side") of something like LFS, and to reason about it from there.
deleted 1y ago
[deleted]
internet_points 1y ago
The page doesn't say it, but git-annex was created by https://www.patreon.com/joeyh https://www.patreon.com/joeyh who also made the wonderful https://joeyh.name/code/moreutils/ https://joeyh.name/code/moreutils/ and https://etckeeper.branchable.com/ https://etckeeper.branchable.com/
- kstrauser 1y ago
  More famously (to me), he was a core Debian contributor to Debian for a couple of decades, starting in 1996. A pretty big chunk of what we think of as Linux came from his keyboard.
kajika91 1y ago
I'm using my self-hosted forgejo. I don't see any benefit of git-annex over LFS so far, I'm not even sure I could setup annex as easily. Digging a little bit I found that git-annex is coded in haskell (not a fan) and seems to be 50% slower (expected from haskell but also only 1 source so far so not really reliable). I don't see appeal of the complexity of the commands, they probably serve a purpose. Once you opened a .gitattributes from git-LFS you pretty much know all you need and you barely need any commands anymore. Also I like how setting up a .gitattribute makes everything transparent the same way .gitignore works. I don't see any equivalent with git-annex. Lastly any "tutorial" or guide about git-annex that won't show me an equivalent of 'git lfs ls-files' will definitely not appeal to me. I'm a big user of 'git status' and 'git lfs ls-files' to check/re-check everything.
- stv0g 1y ago
  There is a soft-fork of Forgejo which adds support for git-annex: https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo
- avar 1y ago
  Annex isn't slow because it's written in Haskell, it tends to be slow because of I/O and paranoia that's warranted as the default behavior in a distributed backup tool. E.g. if you drop something it'll by default check the remotes it has access to for that content in real time, it can be many orders of magnitude faster to use --fast etc., to (somewhat unsafely) skip all that and trust whatever metadata you have a local copy of.
- seanparsons 1y ago
  LFS and git-annex have subtly different use cases in my experience. LFS is for users developing something with git that has large files in the repo like the classic game development example. git-annex is something you'd use to keep some important stuff backed up which happens to involve large files, like a home folder with music or whatever in it. In my case I do the latter.
  - aragilar 1y ago
goku12 1y ago
My only problem with git-annex is Haskell. I don't hate the language itself, but the sheer number of dependencies it has to install is staggering. Many of those dependencies are not used by anything else, or may be incompatible versions when more than one application uses it. The pain is when you install them using the system package manager. Just two Haskell applications - annex and pandoc - are enough to fill your daily updates with may be a dozen little Haskell packages. God forbid you're on a distro that installs from source! It's quite safe to just statically link most, if not all of them directly into the application, even when some of them are shared by other applications. I have seen this complaint repeated a few times. The reply from the Haskelliers seem to be that this is for the fine grained modularity of the library ecosystem. But why do they treat it like everything starts and ends with Haskell? Sometimes, there are other priorities like system administration. None of the other compiled languages have this problem - Rust, Go, Zig, ... Even plain old C and C++ aren't this frustrating with dependencies. I need to clarify that I'm not hostile towards the Haskell language, its ecosystem and its users. It's something I plan to learn myself. But why does this problem exist? And is there a solution?
- aragilar 1y ago
  Which package manager are you using? I've not seen any issues with apt-based systems with Haskell?
  - goku12 1y ago
    I used to have issues on Arch/pacman. Now on ebuilds/Gentoo.
- IsTom 1y ago
  > It's quite safe to just statically link most, if not all of them directly into the application If you're talking about distro's repos, isn't this a matter of distro and package manager policy?
  - goku12 1y ago
    Very likely. But then, why is this an issue with Haskell alone?
andunie 1y ago
I've used this for years, but to me the big selling point was integration with cloud storage providers as a means of backup. That, however, was always flaky and dependent on unmaintained third-party plugins. I think there was also a bug at some point that caused some data inconsistencies, so eventually I stopped. Does anyone know if the situation has improved on that front in the past 5 years?
- matrss 1y ago
  Depends on the cloud storage provider, I think. The best chances are with those that support the more standard protocols like S3, webdav, sftp, etc.. A relatively new development is the special remote built into rclone, which should be better maintained than some other third-party special remotes and provides access to all rclone-supported remotes.
  - andunie 1y ago
    Oh, really? rclone is great, but as a standalone thing it's really annoying to use. I didn't realize until now that it was missing a git-annex integration to be great. Thank you! I'll start using it again.
BrandiATMuhkuh 1y ago
Does this also work if I have data on SharePoint, DropBox, etc. and want to pull them (sync with local machine)? My use case is mostly ETL related, where I want to pull all customers data (enterprise customer) so I can process them. But also keep the data updated, hence pull?
- matrss 1y ago
  In an ideal world the rclone special remote would support git-annex' importtree feature. Then you could periodically run `git annex import <branch>:<subdir> --from <sharepoint/dropbox-remote>` to "pull" from those remotes (it is not really a pull as you aren't fetching version-controlled data from those remotes, rather you are importing from a non-version-controlled source and record the result as a new revision). Unfortunately this is not (yet?) supported I think. But you could also just do something like this: `rclone copy/sync <sharepoint/dropbox-remote>: ./<local-directory> && git annex add ./<local-directory> && git commit -m <message>`.
  - gradientsrneat 1y ago
    git-annex does support rclone as a special remote iirc
    - matrss 1y ago
      Yes it does, but I don't think the special remote supports the importtree feature, which would be necessary for this.
the_absurdist 1y ago
Ironically, I just spent a day last weekend writing my own version control system for large files I dislike git-annex that much. - it converts your files into blobs and bloats your file system - As others have previously alluded, my primary use case is to ensure sync between distributed files, not version them (why would anyone possibly need that??) - You can use AI to build a python based solution that will hash your files and put them into a lookup table, then create some helper methods to sync sources using rclone Far simpler and more efficient methods exist.

225 points by keepamovin 1y ago

Happy to see use cases front and center in command line documentation. They seem to always start with ”obscure command flag that you’ll probably never use”.

ygritte 1y ago

Could this be abused to simulate something like SVN externals? I always found git submodules to be a very bad replacement for that.

fragmede 1y ago

GitHub really embraced the Microsoft-esque NIH with LFS, instead of adopting git-annex.

keepamovin 1y ago
To its absolute detriment Here is a talk by a person who adores it: Yann Büchau: Staying in Control of your Scientific Data with Git Annex https://www.youtube.com/watch?v=IdRUsn-zB2s https://www.youtube.com/watch?v=IdRUsn-zB2s
- codemac 1y ago
  While Yann has built many things with git-annex, we should be clear that the creator of git-annex is relatively singular, Joey Hess.
  - keepamovin 1y ago
    Here is a comment about Joey: https://news.ycombinator.com/item?id=14908529 https://news.ycombinator.com/item?id=14908529 And an interview When power is low, I often hack in the evenings by lantern light. https://usesthis.com/interviews/joey.hess/ https://usesthis.com/interviews/joey.hess/
mathstuf 1y ago
While I also find git-annex more elegant, its cross-platform story is weaker. Note that LFS was originally a collaboration between GitHub and Bitbucket (maybe? Some forge vendor I think). One had the implementation and the other had the name. They met at a Git conference and we have what we have today. My main gripes these days are the woefully inadequate limits GitHub has in place for larger projects. Coupled with the "must have all objects locally to satisfy an arbitrary push", any decently sized developer community will blow the limit fairly quickly. FD: I have contributed to git-lfs.

deleted 1y ago

[deleted]

andrewmcwatters 1y ago

git-annex has some really awkward documentation. You can apparently do, sort of, but not really, the same thing git-fetch-file[1] does, with git-annex: git fetch-file add https://github.com/icculus/physfs.git "**" lib/physfs-main git fetch-file pull `add` creates this at `.git-remote-files`: [file "**"] commit = 9d18d36b5a5207b72f473f05e1b2834e347d8144 target = lib/physfs-main repository = https://github.com/icculus/physfs branch = main But git-annex's documentation goes on and on about a bunch of commands I don't really want to read about, whereas those two lines and that .git-remote-files manifest just told you what git-fetch-file does. [1]: https://github.com/andrewmcwattersandco/git-fetch-file https://github.com/andrewmcwattersandco/git-fetch-file

nolist_policy 1y ago
Not at all. git-annex is for managing large files in git and unlike git-lfs it preserves the distributed nature of git.
keepamovin 1y ago
Here is a guide you might like: https://www.youtube.com/watch?v=p0eVyhv2rbk https://www.youtube.com/watch?v=p0eVyhv2rbk

nolist_policy 1y ago

I use git-annex to manage all my data on all my drives. It automatically keeps track of which files are on which drives, it ensures that there are enough copies and it checksums everything. It works perfectly with offline drives. git-annex can be a bit hard to grasp, so I suggest to create a throw-away repository, following the walkthrough[1] and try things out. See also workflows[2]. [1] https://git-annex.branchable.com/walkthrough/ https://git-annex.branchable.com/walkthrough/ [2] https://git-annex.branchable.com/workflow/ https://git-annex.branchable.com/workflow/

albertzeyer 1y ago
How much data do you have? I'm using git-annex on my photos, and that are around 100k-1M files, several TB of data, on a ZFS. In the beginning, everything was fine, but it starts to become increasingly slow, such that every operation takes several minutes (5-30 mins or so). I wonder a bit whether that is ZFS, or git-annex, or maybe my disk, or sth else.
- riedel 1y ago
  It would be great to have comprehensive benchmarks for git lsf, git annex, dvc and alike. I am also always getting annoyed with one or the other , e.g. due to the hashing overhead, etc. However, in many cases the annoyances come with bad filesystem integration on Windows in my case.
  - rurban 1y ago
    My guess is the windows virus scaner
- warp 1y ago
  My experience is the same, git-annex just doesn't work well with lots of small files. With annexes on slow USB disks, connected to a Raspberry Pi 3 or 4, I'm already annoyed when working with my largest annex (in file count) of 25000 files. However, I mostly use annex as a way to archive stuff and make sure I have enough copies in distinct physical locations. So for photos I now just tar them up with one .tar file per family member per year. This works fine for for me for any data I want to keep safe but don't need to access directly very often.

Munksgaard 1y ago

Git-Annex is a cool piece of technology, but my impression is that it works best for single-user repositories. So for instance, as @nolist_policy described in a sibling comment, managing all your personal files, documents, music, etc. across many different devices. I tried using it for syncing large files in a collaborative repository, and the use of "magic" branches didn't seem to scale well.

geephroh 1y ago
YMMV, but it works for my org. We're an archival institution and have been using git-annex for more than a decade as the storage backend for a digital repository system designed for long-term, robust preservation. Admittedly, we only have 15-20 staff; but +30TB of data, ~750K files (binaries + metadata), across hundreds of collection repos.
- eigengrau 1y ago
  How (if you saw that need) did you address permissions concerns, e.g., around any Git users being able to force drop all files from a backend? Back (long time ago) when I was looking into this, there was no KISS, out-of-the-box way to manage the Git Annex operations a Git user would be allowed to perform. Gitolite (or whatever Git platform of choice) can address access control concerns for regular Git pushes, but there is no way to define policies on Git Annex operations (configuration, storage management). Might not be super hard to create a Gitolite plugin to address these, but ultimately for my use-case it wasn’t worth the effort (I didn’t really need shared Git Annex repos). Do you tackle these concerns somehow? I guess if people don’t interact with your repositories via Git/SSH but only through some custom UI, you might deal with it there.

ttiurani 1y ago

Relevant discussion 9 days ago about the new native git large object promisers in "The future of large files in Git is Git": https://news.ycombinator.com/item?id=44916783 https://news.ycombinator.com/item?id=44916783

avar 1y ago
Thanks, also not-so-relevant, for the reasons I noted in a comment in that thread: https://news.ycombinator.com/item?id=44922405 https://news.ycombinator.com/item?id=44922405 I.e. annex is really in a different problem space than "big files in git", despite the obvious overlap. A good way to think about it is that git-annex is sort of a git-native and distributed solution to the storage problem at the "other side" ("server side") of something like LFS, and to reason about it from there.

deleted 1y ago

[deleted]

internet_points 1y ago

The page doesn't say it, but git-annex was created by https://www.patreon.com/joeyh https://www.patreon.com/joeyh who also made the wonderful https://joeyh.name/code/moreutils/ https://joeyh.name/code/moreutils/ and https://etckeeper.branchable.com/ https://etckeeper.branchable.com/

kstrauser 1y ago
More famously (to me), he was a core Debian contributor to Debian for a couple of decades, starting in 1996. A pretty big chunk of what we think of as Linux came from his keyboard.

kajika91 1y ago

I'm using my self-hosted forgejo. I don't see any benefit of git-annex over LFS so far, I'm not even sure I could setup annex as easily. Digging a little bit I found that git-annex is coded in haskell (not a fan) and seems to be 50% slower (expected from haskell but also only 1 source so far so not really reliable). I don't see appeal of the complexity of the commands, they probably serve a purpose. Once you opened a .gitattributes from git-LFS you pretty much know all you need and you barely need any commands anymore. Also I like how setting up a .gitattribute makes everything transparent the same way .gitignore works. I don't see any equivalent with git-annex. Lastly any "tutorial" or guide about git-annex that won't show me an equivalent of 'git lfs ls-files' will definitely not appeal to me. I'm a big user of 'git status' and 'git lfs ls-files' to check/re-check everything.

stv0g 1y ago
There is a soft-fork of Forgejo which adds support for git-annex: https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo
avar 1y ago
Annex isn't slow because it's written in Haskell, it tends to be slow because of I/O and paranoia that's warranted as the default behavior in a distributed backup tool. E.g. if you drop something it'll by default check the remotes it has access to for that content in real time, it can be many orders of magnitude faster to use --fast etc., to (somewhat unsafely) skip all that and trust whatever metadata you have a local copy of.
seanparsons 1y ago
LFS and git-annex have subtly different use cases in my experience. LFS is for users developing something with git that has large files in the repo like the classic game development example. git-annex is something you'd use to keep some important stuff backed up which happens to involve large files, like a home folder with music or whatever in it. In my case I do the latter.
- aragilar 1y ago

goku12 1y ago

My only problem with git-annex is Haskell. I don't hate the language itself, but the sheer number of dependencies it has to install is staggering. Many of those dependencies are not used by anything else, or may be incompatible versions when more than one application uses it. The pain is when you install them using the system package manager. Just two Haskell applications - annex and pandoc - are enough to fill your daily updates with may be a dozen little Haskell packages. God forbid you're on a distro that installs from source! It's quite safe to just statically link most, if not all of them directly into the application, even when some of them are shared by other applications. I have seen this complaint repeated a few times. The reply from the Haskelliers seem to be that this is for the fine grained modularity of the library ecosystem. But why do they treat it like everything starts and ends with Haskell? Sometimes, there are other priorities like system administration. None of the other compiled languages have this problem - Rust, Go, Zig, ... Even plain old C and C++ aren't this frustrating with dependencies. I need to clarify that I'm not hostile towards the Haskell language, its ecosystem and its users. It's something I plan to learn myself. But why does this problem exist? And is there a solution?

aragilar 1y ago
Which package manager are you using? I've not seen any issues with apt-based systems with Haskell?
- goku12 1y ago
  I used to have issues on Arch/pacman. Now on ebuilds/Gentoo.
IsTom 1y ago
> It's quite safe to just statically link most, if not all of them directly into the application If you're talking about distro's repos, isn't this a matter of distro and package manager policy?
- goku12 1y ago
  Very likely. But then, why is this an issue with Haskell alone?

andunie 1y ago

I've used this for years, but to me the big selling point was integration with cloud storage providers as a means of backup. That, however, was always flaky and dependent on unmaintained third-party plugins. I think there was also a bug at some point that caused some data inconsistencies, so eventually I stopped. Does anyone know if the situation has improved on that front in the past 5 years?

matrss 1y ago
Depends on the cloud storage provider, I think. The best chances are with those that support the more standard protocols like S3, webdav, sftp, etc.. A relatively new development is the special remote built into rclone, which should be better maintained than some other third-party special remotes and provides access to all rclone-supported remotes.
- andunie 1y ago
  Oh, really? rclone is great, but as a standalone thing it's really annoying to use. I didn't realize until now that it was missing a git-annex integration to be great. Thank you! I'll start using it again.

BrandiATMuhkuh 1y ago

Does this also work if I have data on SharePoint, DropBox, etc. and want to pull them (sync with local machine)? My use case is mostly ETL related, where I want to pull all customers data (enterprise customer) so I can process them. But also keep the data updated, hence pull?

matrss 1y ago
In an ideal world the rclone special remote would support git-annex' importtree feature. Then you could periodically run `git annex import <branch>:<subdir> --from <sharepoint/dropbox-remote>` to "pull" from those remotes (it is not really a pull as you aren't fetching version-controlled data from those remotes, rather you are importing from a non-version-controlled source and record the result as a new revision). Unfortunately this is not (yet?) supported I think. But you could also just do something like this: `rclone copy/sync <sharepoint/dropbox-remote>: ./<local-directory> && git annex add ./<local-directory> && git commit -m <message>`.
- gradientsrneat 1y ago
  git-annex does support rclone as a special remote iirc
  - matrss 1y ago
    Yes it does, but I don't think the special remote supports the importtree feature, which would be necessary for this.

the_absurdist 1y ago

Ironically, I just spent a day last weekend writing my own version control system for large files I dislike git-annex that much. - it converts your files into blobs and bloats your file system - As others have previously alluded, my primary use case is to ensure sync between distributed files, not version them (why would anyone possibly need that??) - You can use AI to build a python based solution that will hash your files and put them into a lookup table, then create some helper methods to sync sources using rclone Far simpler and more efficient methods exist.