12 ms·

Avoid Git LFS if possible

143 points by reimbar 5y ago

breck 5y ago
My practice for storing large files with Git is to include the metadata for the large file in a tiny file(s): 1. Type information. Enough to synthesize a fake example. 2. A simple preview. This can be a thumb or video snippet, for example. 3. Checksum and URL of the big file. This way your code can work at compile/test time using the snippet or synthesized data, and you can fetch the actual big data at ship time. You can then also use the best version control tool for the job for the particular big files in question.
- CobrastanJorji 5y ago
  Is this just a manual equivalent of git LFS, or is there some advantage here?
  - breck 5y ago
    It's a design pattern that ensures testability of the system without any dependencies on the big files.
  - theamk 5y ago
    This is pretty superior to git LFS in many aspects: - You have file type and preview that you can use without getting the full thing - You have a custom metadata for each file enforced by your scripts -- for example for archives, you may store the list of files inside. This will allow your CI tests to validate the references into the files without having to download the whole huge thing. - You fully control remote fetch logic. Multiple servers? Migration rules for old revisions? That weird auth scheme that your IT insists on? It is all supported with a bit of code. - You fully control local storage. Do you want a computer-wide shared CAS cache between multiple users? What if you have NAS that most users mount? Or maybe s3fs is your thing? Adding support is easy. The main downside is that you get to do all the tooling and documentation, so I would not recommend this for the smaller teams. Nor would I recommend this for open-source projects. But if your infra team is big enough to support this, you'll definitely have the better experience than generic Git LFS.
- dmm 5y ago
  Tools like git-annex or dvc support similar strategies.
  - remram 5y ago
    Is git-annex still alive? Last time I tried to use it, it was very rough, and the official wiki (that serves as doc + bug tracker) gives database errors trying to create an account. Details: I wanted to have a remote I can push to but anonymous users can only pull from, couldn't piece it together.
    - Izkata 5y ago
      I've always liked the simplicity of git-fat [0]: * Initial setup includes git filter rules so that "git add" automatically uses get-fat for matching files (no need to remember to invoke git-fat when adding/changing files). * It works by rsync'ing to/from the remote. The setup for this is in a single ".gitfat" file, separate from the filter rules. * You do need to run "git fat push" and "git fat pull"; this can probably be automated with hooks. So just offhand without even trying to think about the "right" way to do what you want, the committed ".gitfat" could be to a read-only remote, then you can swap it with your own un-committed file for a push that has an rsync-writeable remote. Also, the whole thing is a single 628-line python file, so worst case it would be easy to tweak it to read something like ".gitfat-push" and not have to manually swap it. [0] https://github.com/jedbrown/git-fat https://github.com/jedbrown/git-fat
      - remram 5y ago
        Thanks I didn't know about this one! It seems to only support rsync though, so using it for public repositories would be difficult.
dheera 5y ago
Okay, so I should avoid it. What is the alternative? I see so many git repos with READMEs saying download this huge pretrained weights file from {Dropbox link, Google drive link, Baidu link, ...} and I don't think that's a very good user experience compared to LFS. LFS itself sucks and should be transparent without having to install it, but it's slightly better than downloading stuff from Dropbox or Google Drive.
- jayd16 5y ago
  According to the article you should use mercurial or PlasticSCM because otherwise you might have to rewrite your history to get to some hypothetical git solution that isn't even on the roadmap. I think I'll stick to LFS.
- rkangel 5y ago
  Some combination of the following two features: Partial clones (https://docs.gitlab.com/ee/topics/git/partial_clone.html https://docs.gitlab.com/ee/topics/git/partial_clone.html) Shallow clones (see the --depth argument: https://linux.die.net/man/1/git-clone https://linux.die.net/man/1/git-clone) The problem with large files is not so much that putting a 1Gb file in Git is a problem. If you just have one revision of it, you get a 1Gb repo, and things run at a reasonable speed. The problem is when you have 10 revisions of the 1Gb file and you end up dealing with 10Gb of data when you only want one, because the default git clone model is to give you the full history of everything since the beginning of time. This is fine for (compressible) text files, less fine for large binary blobs. Git-lfs is a hack and it has caused me pain every time I've used it, despite Gitlab having good support for it. Some of this is more implementation detail - the command line UI has some wierdness to it, there's no clear error if someone doesn't have git-lfs when cloning and so something in your build process down the line breaks with a weird error because you've got a marker file instead of the expected binary blob. Some of it is inherent though - the hardest problem is that we now can't easily mirror the git repo from our internal gitlab to the client's gitlab because the config has to hold the http server address with the blobs in. We have workarounds but they're not fun. The solution is to get over the 'always have the whole repository' thing. This is also useful for massive monorepos because you can clone and checkout just the subfolder you need and not all of everything. I say this, but I haven't yet used partial clones in anger (unlike git-lfs). I have high hopes though, and it's a feature in early days.
SavantIdiot 5y ago
Yep. All of this. I tried using Git LFS for a project and reverted back to links to cloud server for the large binary blobs and hashes on those blobs.
robmsmt 5y ago
Pushing Github past the 100mb limit has to be the most requested feature. Ridiculous that we have to use the fudge that is GitLFS. It just adds complication for a limit that shouldn't be there anyway.
- viraptor 5y ago
  You can use gitlab instead with a few GB limit instead.
  - robmsmt 5y ago
    You can also self host with GOGS / Gitea and no limit but getting my company to move from Github will be a large undertaking. It's not worth it for GitLFS on it's own.
goodcjw2 5y ago
A side topic: is there a concrete reason why github's LFS solution has to be so expensive? IIRC, it's $5 per 50GB per month? That's really a deal breaker to me and wondering whether people actually use LFS at volume will avoid LFS-over-GitHub.
- tux1968 5y ago
  $60 / year for a decent fraction of a hard disk and the associated backup resources, seems pretty fair to me. What price would you expect?
  - moshmosh 5y ago
    It's a very large markup on the small-user retail cost of the basic thing they're providing (web-accessible, access-controlled file storage—see, for example, BackBlaze B2) but that's utterly typical of services that can get away with charging you a "convenience fee" for that sort of thing once you're on their SaaS. 2-3x markup isn't unusual, and that's about what this is, and that's above typical retail—even if GH's not managing the storage and such themselves, they're likely getting an even better (bulk) rate.
    - klodolph 5y ago
      Yes, I would estimate that the markup is more like 5-10x for GitHub LFS.
      - goodcjw2 5y ago
        Exactly. For us we are dealing with lots of large gaming assets and those are burning through those $50 data packs like butters :)
  - Dylan16807 5y ago
    > $60 / year for a decent fraction of a hard disk and the associated backup resources, seems pretty fair to me. What you describe sounds fair to me. The problem is that 50GB is not a decent fraction of a hard disk.
Someone1234 5y ago
All three points are really just the same point repeated three times: That it isn't part of core/official GIT ("stop gap" until official, irreversible to later official solution, and adds complexity that an official version would lack due to extra/third party tooling). I'm frankly surprised GIT hasn't made LFS an official part by now. It fixes the problem, the problem is common and real, and GIT hasn't offered a better alternative. If LFS was made official it would solve this critique, since that is really the only critique here.
- klodolph 5y ago
  > All three points are really just the same point repeated three times Absolutely not. Having worked with Mercurial LFS and Git LFS, the differences seem subtle but they are there. Basically, In Mercurial, LFS is (to an extent) an implementation detail of how you check out a repository. It doesn't mean altering the repository contents itself (the data), it just means altering how you get that data. Contrast with Git LFS, where the data itself must be altered in order to become LFS data, and the "LFS flag" is recorded in history. This is not something that you would solve by upstreaming LFS. You would need to redesign LFS.
- swiley 5y ago
  LFS is definitely outside the scope of git.
  - IshKebab 5y ago
    It shouldn't be though.
    - cerved 5y ago
      Why do you think it should be within the scope of git?
      - IshKebab 5y ago
        Because it's something that lots of people want to use Git for, and because it would work better if it were a core part of Git.
asymptosis 5y ago
Something missing from the list of problems: Git LFS is a http(s) protocol so is problematic at best when you are using Git over ssh[1]. The git-lfs devs obviously don't use ssh, so you get the feeling they are a bit exasperated by this call to support an industry standard protocol which is widely used as part of ecosystems and workflows involving Git. [1] https://github.com/git-lfs/git-lfs/issues/1044 https://github.com/git-lfs/git-lfs/issues/1044
- alkonaut 5y ago
  That issue has come a long way though, already a draft PR! This seems like it could actually happen.
justaguy88 5y ago
> Git LFS is a Stop Gap Solution Build the real thing then..
- klodolph 5y ago
  The author of the article is a Mercurial maintainer, and Mercurial has the "real thing" implemented already (and it has been part of Mercurial since at least 2012, at least in some form). So it's already done, just not for Git.
- formerly_proven 5y ago
  He does. > Since I'm a maintainer of the Mercurial version control tool,
temac 5y ago
If you just don't jump on random tech without good reasons, you already naturally apply this advice. Especially since once you really need it and also wants Git, there is not much alternative (as the author recognizes). In this context, just waiting for a potential "better support for handling of large files" of official Git makes little sense; plus I make the wild prediction that what will actually happen is that it's Git LFS that will (continue to) be improved and used by most people (and maybe even integrated in "official Git"?)
- ziml77 5y ago
  This is what I was thinking too. There's really nothing about Git LFS that should come as a surprise. Yes it rewrites history, but how else are you going to cut bloat from the repo after it's been stuffed in there? And the fact that the file is stored on something completely outside of git is clearly and concisely explained as the main text, directly above the download button, on https://git-lfs.github.com/ https://git-lfs.github.com/ > Git Large File Storage (LFS) replaces large files such as audio samples, videos, datasets, and graphics with text pointers inside Git, while storing the file contents on a remote server like GitHub.com or GitHub Enterprise.
madjam002 5y ago
As much as Git LFS is a bit of a pain, on recent projects I've resorted to committing my node_modules with Yarn 2 to Git using LFS and it works really well. Note that with Yarn 2 you're committing .tar.gz's of packages rather than the JS files themselves, so it lends itself quite well to LFS as there are a smaller number of large files. https://yarnpkg.com/features/zero-installs#how-do-you-reach-this-zero-install-state-youre-advocating-for https://yarnpkg.com/features/zero-installs#how-do-you-reach-... https://yarnpkg.com/features/zero-installs#is-it-different-from-just-checking-in-the-node_modules-folder https://yarnpkg.com/features/zero-installs#is-it-different-f...
- cerved 5y ago
  why are you committing packages?
  - hobofan 5y ago
    I would assume to prevent situations like the left-pad incident.
    - cerved 5y ago
      PMs are made for managing and hosting packages, VCS are made for versioning source-code. If you're checking in packages into VC, you're going against the designs of both your PM and VCS. It's a bad idea. Don't. If you for some reason require redundancy of a package repo, then host your own.
  - madjam002 5y ago
    Because why not? It’s recommended in Yarn 2 and I don’t see there being any downsides with Git LFS as the files stores in Git are essentially pointers.
- devinrhode2 5y ago
  Does yarn2 recommend also using LFS? Do you see any performance improvements when using LFS?
slaymaker1907 5y ago
Is rewriting the history for large repos really that difficult besides coordinating with other contributors? My understanding is that it shouldn't be that much worse than "git gc --aggressive". Yes it is expensive, but it is the sort of thing you can schedule to do overnight or on a weekend.
- alkonaut 5y ago
  The problem I see is that things like commit hashes which are etched in history in bug reports, version tags etc, instantly lose meaning. Whether or not that’s a problem depends on how much of that you have.
- sjansen 5y ago
  The issue is breaking external references. Do you include git SHAs in your bug tracking system? Or perhaps your department wiki links to a specific commit to document lessons learned? Maybe you're using Sentry and find including the git SHA of the build to be invaluable for troubleshooting? For some organizations, rewriting history would be a non-event and for others it would be a major disruption.
  - cookiecaper 5y ago
    Yeah, git is really not a mature or well-designed VCS. The fact that you can trivially lose the supposed permanent reference -- and that it's encouraged as part of several common workflows at that -- should be more than enough to demonstrate this. If you care about history, use a VCS like Fossil.
    - inimino 5y ago
      What is encouraged by Github is not always exactly the same as what is encouraged by the maintainers of git.
    - cerved 5y ago
      the SHA is permanent, you're responsible for backup
CreepGin 5y ago
I've been using Git LFS with several large Unity projects in the past several years. Never really had any problems. It was always just "enable and forget" kind of thing.
- coley 5y ago
  This is my experience as well so far. It took 15-20 minutes to learn about it, install it, and setup configs. Since then I haven't had to think about it once.
- P_I_Staker 5y ago
  Yeah, unless you can avoid large files entirely, or are okay with a separate tool (this pisses of a lot of devs IME), then just don't use git? I don't like that option. I think this is sensational. At this point LFS is a really big deal. I don't think it's going anywhere or that LFS users will be shafted.
hpcjoe 5y ago
Just this past week, git lfs was throwing smudge errors for me. Not really sure what the issue was, I followed the recommendations to disable, pull, and re-enable. And got them again. So I disabled. And left it disabled. Not a solution. This said, the whole git-lfs bit feels like a (bad) afterthought the way its implemented. I'd love to see some significant reduction of complexity (you shouldn't need to do 'git lfs enable', it should be done automatically), and increases in resiliency (sharding into FEC'ed blocks with distributed checksums, etc.) so we don't have to deal with 'bad' files. I was a fan of mercurial before I switched to git ... it was IMO an easier/better system at the time (early 2010s). Not likely to switch now though.
- klodolph 5y ago
  I would say that if you care about good LFS support, that is a sufficient reason to use Mercurial. Harder to find Mercurial hosting these days, though, but I'm not worried that the Mercurial project will die off (since both Facebook and Google use it, in some manner).
  - jfim 5y ago
    Is Facebook still using mercurial? It seems that there was a blog post about it in 2014, but their repo[0] just seems to say that their codebase was originally based on/evolved from mercurial. [0] https://github.com/facebookexperimental/eden https://github.com/facebookexperimental/eden
    - klodolph 5y ago
      I think this is just really the demarcation problem. At what point does it stop being “Mercurial”?
      - neandrake 5y ago
        Losing Facebook as a contributor and community member is the larger loss, regardless of how much their internal vcs resembles or integrates with mercurial.
dwohnitmok 5y ago
git-annex is an interesting alternative the HTTP-first nature of Git LFS and the one-way door bother you. You can remove it after the fact if you don't like it, it supports a ton of protocols, and it's distributed just like git is (you can share the files managed by git-annex among different repos or even among different non-git backends such as S3). The main issue that git-annex does not solve is that, like Git LFS, it's not a part of git proper and it shows in its occasionally clunky integration. By virtue of having more knobs and dials it also potentially has more to learn than Git LFS.
- cesarb 5y ago
  With git-annex you have the same "one-way door" behavior: it replaces large files with a pointer to the content (in git-annex, it's a relative symbolic link which by default encodes the real file's size and hash), which is stored in git-annex's own database.
  - remram 5y ago
    That content can easily be moved in bulk though. It is true that you have to use git-annex command to do so, but this is different from LFS where the complete set of historical files is only stored on the server and can't be moved at all. edit: The article claims it's a "one-way door" because you can't move to an altogether different system without rewriting history, which is true of git-annex. My bad.
  - dwohnitmok 5y ago
    Sort of. The way that the author talks about Mercurial as not having this problem makes me think they're talking about something related but subtly different. In particular, AFAICT, Mercurial requires the exact same thing as what you're pointing out. If you want to completely disable use of largefiles then you still have to run `hg lfconvert` at some point. That also changes your revision history. The "one-way door" as I understand the article to be describing is talking about the additional layer of centralization that Git LFS brings. In particular it's pretty annoying to have to always spin up a full HTTPS server just to be able to have access to your files. There is now always a source of truth that is inconvenient to work around when you might still have the files lying around on a bunch of different hard drives or USB drives. Whereas with git-annex, it is true that without rewriting history, even if you disable git-annex moving forward, you'll still have symlinks in your git history. However, as long as you still have your exact binary files sitting around somewhere, you can always import them back on the fly, so e.g. to move away from git-annex you can just commit the binary files directly to your git directory and then just copy them out to a separate folder whenever you go back to an old commit and re-import them. But perhaps I'm interpreting the author incorrectly, in which case it's hard for me to see how any solution for large files in git would allow you to move back without rewriting history to an ordinary git repository without large file support.
korijn 5y ago
I'm honestly super content with LFS. Wrote our own little API server to hook it up to Azure Blob Storage, never have issues with it. I don't recognize the issues mentioned in the article at all. Our whole team relies on it for years, and it delivers. No problems. Keep up the great work, git-lfs maintainers! Much love.
alkonaut 5y ago
I’m using Git+LFS because my issue tracker, CI/CD etc natively speaks it. Not because it’s in any way superior or even on par with the large file handling of Mercurial (or even SVN to be honest).
Game_Ender 5y ago
The latest version of git has a very similar feature called “partial clones” to what the author describes for Mercurial. All the data is still in your history, no extra tools are needed, but you only fetch the blobs from the server for the commits you checkout. So just like LFS larger blobs not on master are effectively free, but you still grab all the blobs for your current commit. You need server side support, which GitHub and GitLab have, and then a special clone command: git clone --filter=blob:none Some background about the feature is here: https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/ https://github.blog/2020-12-21-get-up-to-speed-with-partial-...
- brandmeyer 5y ago
  This looks too aggressive. The nice thing about git-lfs is that only the binary file type(s) you care about are run through git-lfs. All other ordinary diffable text is treated normally. The blobless clone is going to be ensaddening the next time that I'm examining the history of some source code when I'm hacking away without a network connection.
  - Game_Ender 5y ago
    You can mitigate a bit of this by only ignoring blobs over a certain size like "--filter=blob:limit=256k" which should allow most ordinary text files through. In the end it's the same as LFS though in that without a network examining old commits without a network is a bummer. No free lunch here besides something a bit more complex like git-annex.
    - brandmeyer 5y ago
      Closer, but we're still relying on a proxy for the developer's intent. The gitattributes file provides a version-controlled and review-controlled mechanism to decide exactly which objects get special treatment and which ones don't. Since its a part of the repository itself, you don't have to remind new developers to specify some unusual arguments to git at clone time to avoid a performance disaster. > In the end it's the same as LFS though in that without a network examining old commits without a network is a bummer. Except for a crucial detail: The tools I use to examine history are the log, diff[tool], and blame. All of those tools continue to function normally on an LFS-enabled offline clone. IIUC, `--filter=blob:none` doesn't work at all, and `--filter=blob:limit=256k` is a proxy which almost, but doesn't quite work.
TeeMassive 5y ago
The reason the author provides is in my opinion weak compared to both his alternatives. Sure, lfs contaminates a repository, so do large files, sensitive data removal, and references to packages and package managers that might become obsolete or non-existent in the future. The chance of your project compiling after 15 years, the age of git by the way, are very slim, and the chance that having a entirely compilable history being useful even slimmer. And I think the author's statement about setupping up lfs being hard is exaggerated. It's a handful of command lines that should be in the "welcome at our company" manual anyway. I've used lfs in the past and while it can be misused, as with all other tools, it does the job without too much headaches compared submodules and ignored tracked files.
wbillingsley 5y ago
The solution I've tended to use in classes (where there'll always be some student who hasn't installed LFS) is to store the large files in Artifactory, so they are pulled in at build-time in the same way as libraries. This seemed to me a sensible approach as Artifactory is a repository for binaries (usually, the compiled output of a project). It also seemed to me that the decisions on which versions to retain and when an update to a binary is expected or when that resource is now frozen and a replacement would be a new version is similar to the decision on when a build is a snapshot vs a release.
wokwokwok 5y ago
I despise LFS. I’m sure that if you know how to use it... maybe... you can figure it out. That said; here’s my battle story: Estimate the time it’ll take to move all our repositories from a to b they said. Us: with all branches? Them: just main and develop. Us: you just clone and push to the new origin, it’s not zero but it’s trivial. Weeks later... Yeah. LFS is now banned. LFS is not a distributed version control system; once you use it, a clone is no longer “as good” as the original, because it refers to a LFS server that is independent of your clone. ...also, actually cloning all the LFS content from git lab is both slow and occasionally broken in a way that requires you to restart the clone. :(
- iab 5y ago
  I would rather maintain a handwritten journal of 1s and 0s than use git LFS again
  - toomanyducks 5y ago
    honestly had no idea what git LFS was before seeing this, but okay then
- pooya13 5y ago
  What is your alternative then? Version control binary files and have your repos grow gigabytes?
  - AstralStorm 5y ago
    If you're rewriting files and need the version history, yes. If you're not rewriting the files, also yes. If you don't need the history, put them on a normal web server.
    - jmcnulty 5y ago
      If you need history then still put them on a web server and increment the filenames. Storing large files in a git repo is a misappropriation of the tool. It wasn't designed for that use case.
Aeolun 5y ago
Did he really just try to make the argument that we shouldn’t use LFS because Git will have large file support at some unspecified point in the future? LFS has existed for several years, and as far as I know Git still doesn’t have support for large files. At this point I’m not holding out much hope.
- deleted 5y ago
  [deleted]
- cerved 5y ago
  git supports large files, it just can't track changes in binary files efficiently and if they're large you check in a new blob every modification. If they're just sitting around it's fine, but then why would you have them in VC
  - pooya13 5y ago
    It tracks the changes fine. It’s just that it doesn’t make sense to track changes in a binary.
    - AstralStorm 5y ago
      It does make sense, and there are forms of delta compression particularly suited to various binary formats, which if combined with a unpacker for compressed files make great sense. However, git does not have an efficient binary diff implemented yet. LRzip happens to have such a format preprocessor that would make for exceedingly efficient binary history at cost of being more similar to git pack file than incremental versions. Then again, GitHub in particular sets a very low limit on binary size in version control.
      - rurban 5y ago
        In embedded almost everybody uses efficient binary delta diffs and patching for DFOTA (delta firmware over the air update). Jojodiff exists as GPL and MIT variants. http://jojodiff.sourceforge.net/ http://jojodiff.sourceforge.net/ https://github.com/janjongboom/janpatch https://github.com/janjongboom/janpatch Rsync is also very popular, even if not that efficient. xdelta, bsdiff, BDelta, bdiff are all crap.
KETpXDDzR 5y ago
This opinion only lists issues, not solutions. Sure, they advertise mercurial, but migrating from git to mercurial is unrealistic for many cases. I'd title it: "Why Mercurial is better than git+LFS"
- cerved 5y ago
  The author is detailing the problems wrt git-lfs, why they are problems and how those problems are overcome in a similar technical solution in a similar VCS. I think the original title is fine
ecnahc515 5y ago
You don't need to rewrite history unless you weren't using LFS or accidently committed large files to the repository. Nothing about LFS "requires" rewriting history. Not to mention, many users are paying for a service that provides LFS, and hosting an LFS service isn't crazy hard. It's a file server with a custom API, it's mostly doable using S3 as a backend. It's not like this is crazy complicated stuff.
- cerved 5y ago
  other stuff might require require rewriting history
shabbyrobe 5y ago
Here's another fun one: https://github.com/git-lfs/git-lfs/issues/2434 https://github.com/git-lfs/git-lfs/issues/2434 > Git on Windows client corrupts files > 4Gb It's apparently an upstream issue with Git on Windows, but if you depend on something, you inherit its issues.
acdha 5y ago
This is really overstating the cost of a one-time setup step. History rewriting is only necessary for preexisting projects and you can use things like GitLab’s push rules to ensure that it’s never necessary in the future. I get that a mercurial developer has different preferences but I don’t think that this is an especially effective form of advocacy.
rsync 5y ago
FWIW, rsync.net is currently deploying LFS support such that operations like: ssh user@rsync.net git clone blah ... will properly handle LFS assets, etc. This is in response to several requests we have had for this feature...
sjburt 5y ago
The thing that always rubbed me the wrong way about git-lfs was that they cloned the git-scm.org site design. It's not part of git! [1]https://git-lfs.github.com/ https://git-lfs.github.com/ [2]https://git-scm.com/ https://git-scm.com/
chrisdbanks 5y ago
The main argument here seems to be that we shouldn’t use LFS because Git will have large file support at some unspecified point in the future? Similarly you could argue that we shouldn't use a Covid vaccine because we'll develop a cure in the future..why vaccinate billions of people when we can just treat the 1% of people who get ill? Clearly that argument doesn't work. People need a solution now. Ironically we had to stop using mercurial because it didn't have an LFS alternative even though I prefer it. LFS is definitely not ideal but as a solution to a real world problem, it works. There may be issues around cloning repos and losing history in the future, but those are one off issues where you have to accept the pain, rather than living in pain every day.
pooya13 5y ago
Maybe I am missing the point. What is the alternative this article proposes then?... Also, Git is not central so how can you ever integrate large file support without a separate server?
tpoacher 5y ago
I keep hearing the mantra that "svn is better for large files than git" but never really understood why. To me a large file is a large file; if you make changes, worst case scenario you add the entire new file to the commit, best case you add some sort of binary diff. Does git do the former and svn the latter by any chance?
- lmz 5y ago
  An svn working copy has one version stored locally. A git clone has all versions stored locally. All versions of a large file takes up lots of space.
  - tpoacher 5y ago
    I see. So the idea is not that svn's handling of large files at the repo-level is somehow better than that of a git repo per se, but that it's fine for the (possibly remote) svn repo to take the 'large file' hit, since the (presumably local) wc is disjoint from it, and thus unaffected in terms of local storage. Ok, that makes sense... I've been using a decentralised svn workflow at work for so long, I didn't even think of this :)
    - cerved 5y ago
      I suppose it depends how much disk you have :P
- P_I_Staker 5y ago
  For a sizeable project, or one with lots of binary commits, eg. 1-10 GB sized items per day... that type of thing, you have to use LFS or SVN. Your repo size could easily balloon to terabytes, for every clone. Additionally, I think there's other performance issues, but I don't allow this to happen, so I'm not sure. SVN happily handles terabytes, due to the client server interface. As does LFS. My biggest gripe with LFS is that it turns your distributed tool into a client server one. I kinda wish they had and easy "skip lfs" type option.
thenoblesunfish 5y ago
Good points, but it seems optimistic to assume that git will have good, native, large file support anytime soon. I‘ve been waiting quite a while for git submodules to improve..
sam_goody 5y ago
If you have a hundred images in git, and one cannot be downloaded for any reason, git smudge will not be able to run, and you won't be able to git pull at all. We had an image on AWS go bad, still not sure how. Our devs lost the ability to pull. Disabling LFS could not be done (because of rewriting history). "disable smudge" is not an official option, and none of the hacks work reliably. We finally excluded all images from smudge, and downloaded them with SFTP. Git status shows all the images as having changed, and we are downright unhappy... It would be happy to hear that I just don't know how to use LFS - but even if so, that means the docs are woefully not useful. I want to: 1) Tell LFS to get whatever files it could, and just throw a warning on issues. 2) If image is restored not using LFS, git should still know the file has not been modified (by comparing the checksum or whatever smudge would do).

143 points by reimbar 5y ago

My practice for storing large files with Git is to include the metadata for the large file in a tiny file(s): 1. Type information. Enough to synthesize a fake example. 2. A simple preview. This can be a thumb or video snippet, for example. 3. Checksum and URL of the big file. This way your code can work at compile/test time using the snippet or synthesized data, and you can fetch the actual big data at ship time. You can then also use the best version control tool for the job for the particular big files in question.

CobrastanJorji 5y ago
Is this just a manual equivalent of git LFS, or is there some advantage here?
- breck 5y ago
  It's a design pattern that ensures testability of the system without any dependencies on the big files.
- theamk 5y ago
  This is pretty superior to git LFS in many aspects: - You have file type and preview that you can use without getting the full thing - You have a custom metadata for each file enforced by your scripts -- for example for archives, you may store the list of files inside. This will allow your CI tests to validate the references into the files without having to download the whole huge thing. - You fully control remote fetch logic. Multiple servers? Migration rules for old revisions? That weird auth scheme that your IT insists on? It is all supported with a bit of code. - You fully control local storage. Do you want a computer-wide shared CAS cache between multiple users? What if you have NAS that most users mount? Or maybe s3fs is your thing? Adding support is easy. The main downside is that you get to do all the tooling and documentation, so I would not recommend this for the smaller teams. Nor would I recommend this for open-source projects. But if your infra team is big enough to support this, you'll definitely have the better experience than generic Git LFS.
dmm 5y ago
Tools like git-annex or dvc support similar strategies.
- remram 5y ago
  Is git-annex still alive? Last time I tried to use it, it was very rough, and the official wiki (that serves as doc + bug tracker) gives database errors trying to create an account. Details: I wanted to have a remote I can push to but anonymous users can only pull from, couldn't piece it together.
  - Izkata 5y ago
    I've always liked the simplicity of git-fat [0]: * Initial setup includes git filter rules so that "git add" automatically uses get-fat for matching files (no need to remember to invoke git-fat when adding/changing files). * It works by rsync'ing to/from the remote. The setup for this is in a single ".gitfat" file, separate from the filter rules. * You do need to run "git fat push" and "git fat pull"; this can probably be automated with hooks. So just offhand without even trying to think about the "right" way to do what you want, the committed ".gitfat" could be to a read-only remote, then you can swap it with your own un-committed file for a push that has an rsync-writeable remote. Also, the whole thing is a single 628-line python file, so worst case it would be easy to tweak it to read something like ".gitfat-push" and not have to manually swap it. [0] https://github.com/jedbrown/git-fat https://github.com/jedbrown/git-fat
    - remram 5y ago
      Thanks I didn't know about this one! It seems to only support rsync though, so using it for public repositories would be difficult.

dheera 5y ago

Okay, so I should avoid it. What is the alternative? I see so many git repos with READMEs saying download this huge pretrained weights file from {Dropbox link, Google drive link, Baidu link, ...} and I don't think that's a very good user experience compared to LFS. LFS itself sucks and should be transparent without having to install it, but it's slightly better than downloading stuff from Dropbox or Google Drive.

jayd16 5y ago
According to the article you should use mercurial or PlasticSCM because otherwise you might have to rewrite your history to get to some hypothetical git solution that isn't even on the roadmap. I think I'll stick to LFS.
rkangel 5y ago
Some combination of the following two features: Partial clones (https://docs.gitlab.com/ee/topics/git/partial_clone.html https://docs.gitlab.com/ee/topics/git/partial_clone.html) Shallow clones (see the --depth argument: https://linux.die.net/man/1/git-clone https://linux.die.net/man/1/git-clone) The problem with large files is not so much that putting a 1Gb file in Git is a problem. If you just have one revision of it, you get a 1Gb repo, and things run at a reasonable speed. The problem is when you have 10 revisions of the 1Gb file and you end up dealing with 10Gb of data when you only want one, because the default git clone model is to give you the full history of everything since the beginning of time. This is fine for (compressible) text files, less fine for large binary blobs. Git-lfs is a hack and it has caused me pain every time I've used it, despite Gitlab having good support for it. Some of this is more implementation detail - the command line UI has some wierdness to it, there's no clear error if someone doesn't have git-lfs when cloning and so something in your build process down the line breaks with a weird error because you've got a marker file instead of the expected binary blob. Some of it is inherent though - the hardest problem is that we now can't easily mirror the git repo from our internal gitlab to the client's gitlab because the config has to hold the http server address with the blobs in. We have workarounds but they're not fun. The solution is to get over the 'always have the whole repository' thing. This is also useful for massive monorepos because you can clone and checkout just the subfolder you need and not all of everything. I say this, but I haven't yet used partial clones in anger (unlike git-lfs). I have high hopes though, and it's a feature in early days.

SavantIdiot 5y ago

Yep. All of this. I tried using Git LFS for a project and reverted back to links to cloud server for the large binary blobs and hashes on those blobs.

robmsmt 5y ago

Pushing Github past the 100mb limit has to be the most requested feature. Ridiculous that we have to use the fudge that is GitLFS. It just adds complication for a limit that shouldn't be there anyway.

viraptor 5y ago
You can use gitlab instead with a few GB limit instead.
- robmsmt 5y ago
  You can also self host with GOGS / Gitea and no limit but getting my company to move from Github will be a large undertaking. It's not worth it for GitLFS on it's own.

goodcjw2 5y ago

A side topic: is there a concrete reason why github's LFS solution has to be so expensive? IIRC, it's $5 per 50GB per month? That's really a deal breaker to me and wondering whether people actually use LFS at volume will avoid LFS-over-GitHub.

tux1968 5y ago
$60 / year for a decent fraction of a hard disk and the associated backup resources, seems pretty fair to me. What price would you expect?
- moshmosh 5y ago
  It's a very large markup on the small-user retail cost of the basic thing they're providing (web-accessible, access-controlled file storage—see, for example, BackBlaze B2) but that's utterly typical of services that can get away with charging you a "convenience fee" for that sort of thing once you're on their SaaS. 2-3x markup isn't unusual, and that's about what this is, and that's above typical retail—even if GH's not managing the storage and such themselves, they're likely getting an even better (bulk) rate.
  - klodolph 5y ago
    Yes, I would estimate that the markup is more like 5-10x for GitHub LFS.
    - goodcjw2 5y ago
      Exactly. For us we are dealing with lots of large gaming assets and those are burning through those $50 data packs like butters :)
- Dylan16807 5y ago
  > $60 / year for a decent fraction of a hard disk and the associated backup resources, seems pretty fair to me. What you describe sounds fair to me. The problem is that 50GB is not a decent fraction of a hard disk.

Someone1234 5y ago

All three points are really just the same point repeated three times: That it isn't part of core/official GIT ("stop gap" until official, irreversible to later official solution, and adds complexity that an official version would lack due to extra/third party tooling). I'm frankly surprised GIT hasn't made LFS an official part by now. It fixes the problem, the problem is common and real, and GIT hasn't offered a better alternative. If LFS was made official it would solve this critique, since that is really the only critique here.

klodolph 5y ago
> All three points are really just the same point repeated three times Absolutely not. Having worked with Mercurial LFS and Git LFS, the differences seem subtle but they are there. Basically, In Mercurial, LFS is (to an extent) an implementation detail of how you check out a repository. It doesn't mean altering the repository contents itself (the data), it just means altering how you get that data. Contrast with Git LFS, where the data itself must be altered in order to become LFS data, and the "LFS flag" is recorded in history. This is not something that you would solve by upstreaming LFS. You would need to redesign LFS.
swiley 5y ago
LFS is definitely outside the scope of git.
- IshKebab 5y ago
  It shouldn't be though.
  - cerved 5y ago
    Why do you think it should be within the scope of git?
    - IshKebab 5y ago
      Because it's something that lots of people want to use Git for, and because it would work better if it were a core part of Git.

asymptosis 5y ago

Something missing from the list of problems: Git LFS is a http(s) protocol so is problematic at best when you are using Git over ssh[1]. The git-lfs devs obviously don't use ssh, so you get the feeling they are a bit exasperated by this call to support an industry standard protocol which is widely used as part of ecosystems and workflows involving Git. [1] https://github.com/git-lfs/git-lfs/issues/1044 https://github.com/git-lfs/git-lfs/issues/1044

alkonaut 5y ago
That issue has come a long way though, already a draft PR! This seems like it could actually happen.

justaguy88 5y ago

> Git LFS is a Stop Gap Solution Build the real thing then..

klodolph 5y ago
The author of the article is a Mercurial maintainer, and Mercurial has the "real thing" implemented already (and it has been part of Mercurial since at least 2012, at least in some form). So it's already done, just not for Git.
formerly_proven 5y ago
He does. > Since I'm a maintainer of the Mercurial version control tool,

temac 5y ago

If you just don't jump on random tech without good reasons, you already naturally apply this advice. Especially since once you really need it and also wants Git, there is not much alternative (as the author recognizes). In this context, just waiting for a potential "better support for handling of large files" of official Git makes little sense; plus I make the wild prediction that what will actually happen is that it's Git LFS that will (continue to) be improved and used by most people (and maybe even integrated in "official Git"?)

ziml77 5y ago
This is what I was thinking too. There's really nothing about Git LFS that should come as a surprise. Yes it rewrites history, but how else are you going to cut bloat from the repo after it's been stuffed in there? And the fact that the file is stored on something completely outside of git is clearly and concisely explained as the main text, directly above the download button, on https://git-lfs.github.com/ https://git-lfs.github.com/ > Git Large File Storage (LFS) replaces large files such as audio samples, videos, datasets, and graphics with text pointers inside Git, while storing the file contents on a remote server like GitHub.com or GitHub Enterprise.

madjam002 5y ago

As much as Git LFS is a bit of a pain, on recent projects I've resorted to committing my node_modules with Yarn 2 to Git using LFS and it works really well. Note that with Yarn 2 you're committing .tar.gz's of packages rather than the JS files themselves, so it lends itself quite well to LFS as there are a smaller number of large files. https://yarnpkg.com/features/zero-installs#how-do-you-reach-this-zero-install-state-youre-advocating-for https://yarnpkg.com/features/zero-installs#how-do-you-reach-... https://yarnpkg.com/features/zero-installs#is-it-different-from-just-checking-in-the-node_modules-folder https://yarnpkg.com/features/zero-installs#is-it-different-f...

cerved 5y ago
why are you committing packages?
- hobofan 5y ago
  I would assume to prevent situations like the left-pad incident.
  - cerved 5y ago
    PMs are made for managing and hosting packages, VCS are made for versioning source-code. If you're checking in packages into VC, you're going against the designs of both your PM and VCS. It's a bad idea. Don't. If you for some reason require redundancy of a package repo, then host your own.
- madjam002 5y ago
  Because why not? It’s recommended in Yarn 2 and I don’t see there being any downsides with Git LFS as the files stores in Git are essentially pointers.
devinrhode2 5y ago
Does yarn2 recommend also using LFS? Do you see any performance improvements when using LFS?

slaymaker1907 5y ago

Is rewriting the history for large repos really that difficult besides coordinating with other contributors? My understanding is that it shouldn't be that much worse than "git gc --aggressive". Yes it is expensive, but it is the sort of thing you can schedule to do overnight or on a weekend.

alkonaut 5y ago
The problem I see is that things like commit hashes which are etched in history in bug reports, version tags etc, instantly lose meaning. Whether or not that’s a problem depends on how much of that you have.
sjansen 5y ago
The issue is breaking external references. Do you include git SHAs in your bug tracking system? Or perhaps your department wiki links to a specific commit to document lessons learned? Maybe you're using Sentry and find including the git SHA of the build to be invaluable for troubleshooting? For some organizations, rewriting history would be a non-event and for others it would be a major disruption.
- cookiecaper 5y ago
  Yeah, git is really not a mature or well-designed VCS. The fact that you can trivially lose the supposed permanent reference -- and that it's encouraged as part of several common workflows at that -- should be more than enough to demonstrate this. If you care about history, use a VCS like Fossil.
  - inimino 5y ago
    What is encouraged by Github is not always exactly the same as what is encouraged by the maintainers of git.
  - cerved 5y ago
    the SHA is permanent, you're responsible for backup

CreepGin 5y ago

I've been using Git LFS with several large Unity projects in the past several years. Never really had any problems. It was always just "enable and forget" kind of thing.

coley 5y ago
This is my experience as well so far. It took 15-20 minutes to learn about it, install it, and setup configs. Since then I haven't had to think about it once.
P_I_Staker 5y ago
Yeah, unless you can avoid large files entirely, or are okay with a separate tool (this pisses of a lot of devs IME), then just don't use git? I don't like that option. I think this is sensational. At this point LFS is a really big deal. I don't think it's going anywhere or that LFS users will be shafted.

hpcjoe 5y ago

Just this past week, git lfs was throwing smudge errors for me. Not really sure what the issue was, I followed the recommendations to disable, pull, and re-enable. And got them again. So I disabled. And left it disabled. Not a solution. This said, the whole git-lfs bit feels like a (bad) afterthought the way its implemented. I'd love to see some significant reduction of complexity (you shouldn't need to do 'git lfs enable', it should be done automatically), and increases in resiliency (sharding into FEC'ed blocks with distributed checksums, etc.) so we don't have to deal with 'bad' files. I was a fan of mercurial before I switched to git ... it was IMO an easier/better system at the time (early 2010s). Not likely to switch now though.

klodolph 5y ago
I would say that if you care about good LFS support, that is a sufficient reason to use Mercurial. Harder to find Mercurial hosting these days, though, but I'm not worried that the Mercurial project will die off (since both Facebook and Google use it, in some manner).
- jfim 5y ago
  Is Facebook still using mercurial? It seems that there was a blog post about it in 2014, but their repo[0] just seems to say that their codebase was originally based on/evolved from mercurial. [0] https://github.com/facebookexperimental/eden https://github.com/facebookexperimental/eden
  - klodolph 5y ago
    I think this is just really the demarcation problem. At what point does it stop being “Mercurial”?
    - neandrake 5y ago
      Losing Facebook as a contributor and community member is the larger loss, regardless of how much their internal vcs resembles or integrates with mercurial.

dwohnitmok 5y ago

git-annex is an interesting alternative the HTTP-first nature of Git LFS and the one-way door bother you. You can remove it after the fact if you don't like it, it supports a ton of protocols, and it's distributed just like git is (you can share the files managed by git-annex among different repos or even among different non-git backends such as S3). The main issue that git-annex does not solve is that, like Git LFS, it's not a part of git proper and it shows in its occasionally clunky integration. By virtue of having more knobs and dials it also potentially has more to learn than Git LFS.

cesarb 5y ago
With git-annex you have the same "one-way door" behavior: it replaces large files with a pointer to the content (in git-annex, it's a relative symbolic link which by default encodes the real file's size and hash), which is stored in git-annex's own database.
- remram 5y ago
  That content can easily be moved in bulk though. It is true that you have to use git-annex command to do so, but this is different from LFS where the complete set of historical files is only stored on the server and can't be moved at all. edit: The article claims it's a "one-way door" because you can't move to an altogether different system without rewriting history, which is true of git-annex. My bad.
- dwohnitmok 5y ago
  Sort of. The way that the author talks about Mercurial as not having this problem makes me think they're talking about something related but subtly different. In particular, AFAICT, Mercurial requires the exact same thing as what you're pointing out. If you want to completely disable use of largefiles then you still have to run `hg lfconvert` at some point. That also changes your revision history. The "one-way door" as I understand the article to be describing is talking about the additional layer of centralization that Git LFS brings. In particular it's pretty annoying to have to always spin up a full HTTPS server just to be able to have access to your files. There is now always a source of truth that is inconvenient to work around when you might still have the files lying around on a bunch of different hard drives or USB drives. Whereas with git-annex, it is true that without rewriting history, even if you disable git-annex moving forward, you'll still have symlinks in your git history. However, as long as you still have your exact binary files sitting around somewhere, you can always import them back on the fly, so e.g. to move away from git-annex you can just commit the binary files directly to your git directory and then just copy them out to a separate folder whenever you go back to an old commit and re-import them. But perhaps I'm interpreting the author incorrectly, in which case it's hard for me to see how any solution for large files in git would allow you to move back without rewriting history to an ordinary git repository without large file support.

korijn 5y ago

I'm honestly super content with LFS. Wrote our own little API server to hook it up to Azure Blob Storage, never have issues with it. I don't recognize the issues mentioned in the article at all. Our whole team relies on it for years, and it delivers. No problems. Keep up the great work, git-lfs maintainers! Much love.

alkonaut 5y ago

I’m using Git+LFS because my issue tracker, CI/CD etc natively speaks it. Not because it’s in any way superior or even on par with the large file handling of Mercurial (or even SVN to be honest).

Game_Ender 5y ago

The latest version of git has a very similar feature called “partial clones” to what the author describes for Mercurial. All the data is still in your history, no extra tools are needed, but you only fetch the blobs from the server for the commits you checkout. So just like LFS larger blobs not on master are effectively free, but you still grab all the blobs for your current commit. You need server side support, which GitHub and GitLab have, and then a special clone command: git clone --filter=blob:none Some background about the feature is here: https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/ https://github.blog/2020-12-21-get-up-to-speed-with-partial-...

brandmeyer 5y ago
This looks too aggressive. The nice thing about git-lfs is that only the binary file type(s) you care about are run through git-lfs. All other ordinary diffable text is treated normally. The blobless clone is going to be ensaddening the next time that I'm examining the history of some source code when I'm hacking away without a network connection.
- Game_Ender 5y ago
  You can mitigate a bit of this by only ignoring blobs over a certain size like "--filter=blob:limit=256k" which should allow most ordinary text files through. In the end it's the same as LFS though in that without a network examining old commits without a network is a bummer. No free lunch here besides something a bit more complex like git-annex.
  - brandmeyer 5y ago
    Closer, but we're still relying on a proxy for the developer's intent. The gitattributes file provides a version-controlled and review-controlled mechanism to decide exactly which objects get special treatment and which ones don't. Since its a part of the repository itself, you don't have to remind new developers to specify some unusual arguments to git at clone time to avoid a performance disaster. > In the end it's the same as LFS though in that without a network examining old commits without a network is a bummer. Except for a crucial detail: The tools I use to examine history are the log, diff[tool], and blame. All of those tools continue to function normally on an LFS-enabled offline clone. IIUC, `--filter=blob:none` doesn't work at all, and `--filter=blob:limit=256k` is a proxy which almost, but doesn't quite work.

TeeMassive 5y ago

The reason the author provides is in my opinion weak compared to both his alternatives. Sure, lfs contaminates a repository, so do large files, sensitive data removal, and references to packages and package managers that might become obsolete or non-existent in the future. The chance of your project compiling after 15 years, the age of git by the way, are very slim, and the chance that having a entirely compilable history being useful even slimmer. And I think the author's statement about setupping up lfs being hard is exaggerated. It's a handful of command lines that should be in the "welcome at our company" manual anyway. I've used lfs in the past and while it can be misused, as with all other tools, it does the job without too much headaches compared submodules and ignored tracked files.

wbillingsley 5y ago

The solution I've tended to use in classes (where there'll always be some student who hasn't installed LFS) is to store the large files in Artifactory, so they are pulled in at build-time in the same way as libraries. This seemed to me a sensible approach as Artifactory is a repository for binaries (usually, the compiled output of a project). It also seemed to me that the decisions on which versions to retain and when an update to a binary is expected or when that resource is now frozen and a replacement would be a new version is similar to the decision on when a build is a snapshot vs a release.

wokwokwok 5y ago

I despise LFS. I’m sure that if you know how to use it... maybe... you can figure it out. That said; here’s my battle story: Estimate the time it’ll take to move all our repositories from a to b they said. Us: with all branches? Them: just main and develop. Us: you just clone and push to the new origin, it’s not zero but it’s trivial. Weeks later... Yeah. LFS is now banned. LFS is not a distributed version control system; once you use it, a clone is no longer “as good” as the original, because it refers to a LFS server that is independent of your clone. ...also, actually cloning all the LFS content from git lab is both slow and occasionally broken in a way that requires you to restart the clone. :(

iab 5y ago
I would rather maintain a handwritten journal of 1s and 0s than use git LFS again
- toomanyducks 5y ago
  honestly had no idea what git LFS was before seeing this, but okay then
pooya13 5y ago
What is your alternative then? Version control binary files and have your repos grow gigabytes?
- AstralStorm 5y ago
  If you're rewriting files and need the version history, yes. If you're not rewriting the files, also yes. If you don't need the history, put them on a normal web server.
  - jmcnulty 5y ago
    If you need history then still put them on a web server and increment the filenames. Storing large files in a git repo is a misappropriation of the tool. It wasn't designed for that use case.

Aeolun 5y ago

Did he really just try to make the argument that we shouldn’t use LFS because Git will have large file support at some unspecified point in the future? LFS has existed for several years, and as far as I know Git still doesn’t have support for large files. At this point I’m not holding out much hope.

deleted 5y ago
[deleted]
cerved 5y ago
git supports large files, it just can't track changes in binary files efficiently and if they're large you check in a new blob every modification. If they're just sitting around it's fine, but then why would you have them in VC
- pooya13 5y ago
  It tracks the changes fine. It’s just that it doesn’t make sense to track changes in a binary.
  - AstralStorm 5y ago
    It does make sense, and there are forms of delta compression particularly suited to various binary formats, which if combined with a unpacker for compressed files make great sense. However, git does not have an efficient binary diff implemented yet. LRzip happens to have such a format preprocessor that would make for exceedingly efficient binary history at cost of being more similar to git pack file than incremental versions. Then again, GitHub in particular sets a very low limit on binary size in version control.
    - rurban 5y ago
      In embedded almost everybody uses efficient binary delta diffs and patching for DFOTA (delta firmware over the air update). Jojodiff exists as GPL and MIT variants. http://jojodiff.sourceforge.net/ http://jojodiff.sourceforge.net/ https://github.com/janjongboom/janpatch https://github.com/janjongboom/janpatch Rsync is also very popular, even if not that efficient. xdelta, bsdiff, BDelta, bdiff are all crap.

KETpXDDzR 5y ago

This opinion only lists issues, not solutions. Sure, they advertise mercurial, but migrating from git to mercurial is unrealistic for many cases. I'd title it: "Why Mercurial is better than git+LFS"

cerved 5y ago
The author is detailing the problems wrt git-lfs, why they are problems and how those problems are overcome in a similar technical solution in a similar VCS. I think the original title is fine

ecnahc515 5y ago

You don't need to rewrite history unless you weren't using LFS or accidently committed large files to the repository. Nothing about LFS "requires" rewriting history. Not to mention, many users are paying for a service that provides LFS, and hosting an LFS service isn't crazy hard. It's a file server with a custom API, it's mostly doable using S3 as a backend. It's not like this is crazy complicated stuff.

cerved 5y ago
other stuff might require require rewriting history

shabbyrobe 5y ago

Here's another fun one: https://github.com/git-lfs/git-lfs/issues/2434 https://github.com/git-lfs/git-lfs/issues/2434 > Git on Windows client corrupts files > 4Gb It's apparently an upstream issue with Git on Windows, but if you depend on something, you inherit its issues.

acdha 5y ago

This is really overstating the cost of a one-time setup step. History rewriting is only necessary for preexisting projects and you can use things like GitLab’s push rules to ensure that it’s never necessary in the future. I get that a mercurial developer has different preferences but I don’t think that this is an especially effective form of advocacy.

rsync 5y ago

FWIW, rsync.net is currently deploying LFS support such that operations like: ssh user@rsync.net git clone blah ... will properly handle LFS assets, etc. This is in response to several requests we have had for this feature...

sjburt 5y ago

The thing that always rubbed me the wrong way about git-lfs was that they cloned the git-scm.org site design. It's not part of git! [1]https://git-lfs.github.com/ https://git-lfs.github.com/ [2]https://git-scm.com/ https://git-scm.com/

chrisdbanks 5y ago

The main argument here seems to be that we shouldn’t use LFS because Git will have large file support at some unspecified point in the future? Similarly you could argue that we shouldn't use a Covid vaccine because we'll develop a cure in the future..why vaccinate billions of people when we can just treat the 1% of people who get ill? Clearly that argument doesn't work. People need a solution now. Ironically we had to stop using mercurial because it didn't have an LFS alternative even though I prefer it. LFS is definitely not ideal but as a solution to a real world problem, it works. There may be issues around cloning repos and losing history in the future, but those are one off issues where you have to accept the pain, rather than living in pain every day.

pooya13 5y ago

Maybe I am missing the point. What is the alternative this article proposes then?... Also, Git is not central so how can you ever integrate large file support without a separate server?

tpoacher 5y ago

I keep hearing the mantra that "svn is better for large files than git" but never really understood why. To me a large file is a large file; if you make changes, worst case scenario you add the entire new file to the commit, best case you add some sort of binary diff. Does git do the former and svn the latter by any chance?

lmz 5y ago
An svn working copy has one version stored locally. A git clone has all versions stored locally. All versions of a large file takes up lots of space.
- tpoacher 5y ago
  I see. So the idea is not that svn's handling of large files at the repo-level is somehow better than that of a git repo per se, but that it's fine for the (possibly remote) svn repo to take the 'large file' hit, since the (presumably local) wc is disjoint from it, and thus unaffected in terms of local storage. Ok, that makes sense... I've been using a decentralised svn workflow at work for so long, I didn't even think of this :)
  - cerved 5y ago
    I suppose it depends how much disk you have :P
P_I_Staker 5y ago
For a sizeable project, or one with lots of binary commits, eg. 1-10 GB sized items per day... that type of thing, you have to use LFS or SVN. Your repo size could easily balloon to terabytes, for every clone. Additionally, I think there's other performance issues, but I don't allow this to happen, so I'm not sure. SVN happily handles terabytes, due to the client server interface. As does LFS. My biggest gripe with LFS is that it turns your distributed tool into a client server one. I kinda wish they had and easy "skip lfs" type option.

thenoblesunfish 5y ago

Good points, but it seems optimistic to assume that git will have good, native, large file support anytime soon. I‘ve been waiting quite a while for git submodules to improve..

sam_goody 5y ago

If you have a hundred images in git, and one cannot be downloaded for any reason, git smudge will not be able to run, and you won't be able to git pull at all. We had an image on AWS go bad, still not sure how. Our devs lost the ability to pull. Disabling LFS could not be done (because of rewriting history). "disable smudge" is not an official option, and none of the hacks work reliably. We finally excluded all images from smudge, and downloaded them with SFTP. Git status shows all the images as having changed, and we are downright unhappy... It would be happy to hear that I just don't know how to use LFS - but even if so, that means the docs are woefully not useful. I want to: 1) Tell LFS to get whatever files it could, and just throw a warning on issues. 2) If image is restored not using LFS, git should still know the file has not been modified (by comparing the checksum or whatever smudge would do).