7 ms·

The future of large files in Git is Git

557 points by thcipriani 1y ago

matheusmoreira 1y ago
As it should be! If it's not native to git, it's not worth using. I'm glad these issues are finally being solved. These new features are pretty awesome too. Especially separate large object remotes. They will probably enable git to be used for even more things than it's already being used for. They will enable new ways to work with git.
Affric 1y ago
Incredible. Nice to see some Microsoft and Google emails contributing.
- Maxious 1y ago
  Microsoft uses git for office development https://danielsada.tech/blog/carreer-part-7-how-office-moved-to-git-and-i-loved-devex/ https://danielsada.tech/blog/carreer-part-7-how-office-moved... Google has android and chromium as well as Git+Gerrit-on-Borg https://opensource.google/documentation/reference/releasing/hosting/ https://opensource.google/documentation/reference/releasing/...
als0 1y ago
10 years late is better than never.
tombert 1y ago
Is Git ever going to get proper support for binary files? I’ve never used it for anything serious but my understanding is that Mercurial handles binary files better? Like it supports binary diffs if I understand correctly. Any reason Git couldn’t get that?
- ks2048 1y ago
  I'm not sure binary diffs are the problem - e.g. for storing images or MP3s, binary diffs are usually worse than nothing.
  - digikata 1y ago
    I would think that git would need a parallel storage scheme for binaries. Something that does binary chunking and deduplication between revisions, but keeps the same merkle referencing scheme as everything else.
    - tempay 1y ago
      > binary chunking and deduplication Are there many binaries that people would store in git where this would actually help? I assume most files end up with compression or some other form of randomization between revisions making deduplication futile.
      - hinkley 1y ago
        It would likely require more tooling.
      - digikata 1y ago
        I don't know, it's all probability in the dataset that makes one optimization strategy better over another. Git annex iirc does file level dedupe. That would take care of most of the problem if you're storing binaries that are compressed or encrypted. It's a lot of work to go beyond that, and probably one reason no one has bothered with git yet. But borg and restic both do chunked dedupe I think.
      - adastra22 1y ago
        A lot in the game and visual art industries.
      - zigzag312 1y ago
        2-3x reduction in repository size compared to Git LFS in this test: https://xethub.com/blog/benchmarking-the-modern-development-experience https://xethub.com/blog/benchmarking-the-modern-development-...
    - zigzag312 1y ago
      Xet uses block level deduplication.
- firesteelrain 1y ago
  A lot of people use Perforce Helix and others use Plastic SCM. That’s been my experience for like large binary assets with git-like functionality
  - tom_ 1y ago
    I didn't enjoy using Plastic, but Perforce is ok (not to say that it's perfect - I miss a lot of git stuff). It does have no problems with lots of data though! This article moans about the overhead of a 25 MB png file... it's been a long time since i worked on a serious project where the head revision is less than 25 GB. Typical daily churn would be 2.5 GB+. (It's been even longer since i used svn in anger, but maybe it could work too. It has file locking, and local storage cost is proportional to size of head revision. It was manageable enough with a 2 GB head revision. Metadata access speed was always terrible though, which was tedious.)
    - firesteelrain 1y ago
      SVN should be able to handle large files no issue imho
- brucehoult 1y ago
  All files in git are binary files. All deltas between versions are binary diffs. Git has always handled large (including large binary) files just fine. What it doesn't like is files where a conceptually minor change changes the entire file, for example compressed or encrypted files. The only somewhat valid complaint is that if someone once committed a large file and then it was later deleted (maybe minutes later, maybe years later) then it is in the repo and in everyone's checkouts forever. Which applies equally to small and to large files, but large ones have more impact. That's the whole point of a version control system. To preserve the history, allowing earlier versions to be recreated. The better solution would be to have better review of changes pushed to the master repo, including having unreviewed changes in separate, potentially sacrificial, repos until approved.
- afiori 1y ago
  My understanding is that git diff algorithms require a file to be segmentable (eg text files are split line-wise) and there is no general segmentation strategy for binary blobs. But a good segmentation is only good for better compression and nicer diff, git could do byte wise diffs with no issues, so I wonder why doesn't git use customizable segmentation strategies where it calls external tools based on file type (eg a rust thingy for rust file etc, or a PNG thingy for PNG files). At worst the tool would return either a single segment for the entire file or the byte wise split which would work anyway
  - joeyh 1y ago
    A common misconception. git has always used binary deltas for pack files. Consider that git tree objects are themselves not text files, and git needs efficiently store slightly modified versions of the same tree.
    - afiori 1y ago
      so are pack files diffed byte-wise?
jauer 1y ago
TFA asserts that Git LFS is bad for several reasons including because proprietary with vendor lock-in which I don't think is fair to claim. GitHub provided an open client and server which negates that. LFS does break disconnected/offline/sneakernet operations which wasn't mentioned and is not awesome, but those are niche workflows. It sounds like that would also be broken with promisors. The `git partial clone` examples are cool! The description of Large Object Promisors makes it sound like they take the client-side complexity in LFS, move it server-side, and then increases the complexity? Instead of the client uploading to a git server and to a LFS server it uploads to a git server which in turn uploads to an object store, but the client will download directly from the object store? Obviously different tradeoffs there. I'm curious how often people will get bit by uploading to public git servers which upload to hidden promisor remotes.
- IshKebab 1y ago
  LFS is bad. The server implementations suck. It conflates object contents with the storage method. It's opt-in, in a terrible way - if you do the obvious thing you get tiny text files instead of the files you actually want. I dunno if their solution is any better but it's fairly unarguable that LFS is bad.
  - jayd16 1y ago
    It does seem like this proposal has exactly the same issue. Unless this new method blocks cloning when unable to access the promisors, you'll end up with similar problems of broken large files.
    - cowsandmilk 1y ago
      How so? This proposal doesn’t require you to run `git lfs install` to get the correct files…
      - jayd16 1y ago
        If the architecture is irrelevant and it's just a matter of turning it on by default they could have done that with LFS long ago.
glitchc 1y ago
No. This is not a solution. While git LFS is just a kludge for now, writing a filter argument during the clone operation is not the long-term solution either. Git clone is the very first command most people will run when learning how to use git. Emphasized for effect: the very first command. Will they remember to write the filter? Maybe, if the tutorial to the cool codebase they're trying to access mentions it. Maybe not. What happens if they don't? It may take a long time without any obvious indication. And if they do? The cloned repo might not be compilable/usable since the blobs are missing. Say they do get it right. Will they understand it? Most likely not. We are exposing the inner workings of git on the very first command they learn. What's a blob? Why do I need to filter on it? Where are blobs stored? It's classic abstraction leakage. This is a solved problem: Rsync does it. Just port the bloody implementation over. It does mean supporting alternative representations or moving away from blobs altogether, which git maintainers seem unwilling to do.
- IshKebab 1y ago
  I totally agree. This follows a long tradition of Git "fixing" things by adding a flag that 99% of users won't ever discover. They never fix the defaults. And yes, you can fix defaults without breaking backwards compatibility.
  - Jenk 1y ago
    > They never fix the defaults Not strictly true. They did change the default push behaviour from "matching" to "simple" in Git 2.0.
    - hinkley 1y ago
      So what was the second time the stopped watch was right? I agree with GP. The git community is very fond of doing checkbox fixes for team problems that aren’t or can’t be set as defaults and so require constant user intervention to work. See also some of the sparse checkout systems and adding notes to commits after the fact. They only work if you turn every pull and push into a flurry of activity. Which means they will never work from your IDE. Those are non fixes that pollute the space for actual fixes.
goneri 1y ago
git-annex is a good alternative to the solution of Githu, and it supports different storage backends. I'm actually surprised it's not more popular.
- deleted 1y ago
  [deleted]
- alchemist1e9 1y ago
  Was looking for a mention of git-annex and I completely agree. I’ve used it extensively and have found it works really well. Any ideas why it isn’t more popular and more well known?
  - cesarb 1y ago
    > Any ideas why it isn’t more popular and more well known? While git-annex works very well on Unix-style systems with Unix-style filesystems, it heavily depends on symbolic links, which do not exist on filesystems like exFAT, and are problematic on Windows (AFAIK, you have to be an administrator, or enable an obscure group policy). It has a degraded mode for these filesystems, but uses twice the disk space in that mode, and AFAIK loses some features.
  - avar 1y ago
    I use it, and love it. But it's not intended for or good at (without forcing a square peg into a round hole) the sort of thing LFS and promisors are for, which is a public project with binary assets. git-annex is really for (and shines at) a private backup solution where you'd like to have N copies of some data around on various storage devices, track the history of each copy, ensure that you have at least N copies etc. Each repository gets a UUID, and each tracked file has a SHA-256 hash. There's a branch which has a timestamp and repo UUID to SHA-256 mapping, if you have 10 repos that file will have (at least) 10 entries. You can "trust" different repositories to different degrees, e.g. if you're storing a file on both some RAID'd storage server, or an old portable HD you're keeping in a desk drawer. This really doesn't scale for a public project. E.g. I have a repository that I back up my photos and videos in, that repository has ~700 commits, and ~6000 commits to the metadata "git-annex" branch, pretty close to a 1:10 ratio. There's an exhaustive history of every file movement that's ever occurred on the 10 storage devices I've ever used for that repository. Now imagine doing all that on a project used by more than one person. All other solutions to tracking large files along with a git repository forgo all this complexity in favor of basically saying "just get the rest where you cloned me from, they'll have it!".
forrestthewoods 1y ago
Git is fundamentally broken and bad. Almost all projects are defacto centralized. Your project is not Linux. A good version control system would support petabyte scale history and terabyte scale clones via sparse virtual filesystem. Git’s design is just bad for almost all projects that aren’t Linux. (I know this will get downvoted. But most modern programmers have never used anything but Git and so they don’t realize their tool is actually quite bad! It’s a shame.)
- matheusmoreira 1y ago
  Completely disagree. Git is fundamentally functional and good. All projects are local and decentralized, and any "centralization" is in fact just git hosting services, of which there are many options which are not even mutually exclusive.
  - the_arun 1y ago
    Are you missing the central hosting services provide a good backup plan for your locally hosted git?
    - matheusmoreira 1y ago
      I agree! They are excellent git backup services. I use several of them: github, codeberg, gitlab, sourcehut. I can easily set up remotes to push to all of them at once. I also have copies of my important repositories on all my personal computers, including my phone. This is only possible because git is decentralized. Claiming that git is centralized is complete falsehood.
  - compiler-guy 1y ago
    Got works fine and is solid and well enough known to be a reasonable choice for most people. But I encourage everyone to try out a few alternatives (and adopt their workflows at least for a while). I have no idea if you have or not. But fine has never used the alternatives, one doesn’t really know just how nice things can be. Or, even if you still find fit to be your preferred can, having an alternative experience can open you to other possibilities and ways of working. Just like everyone should try a couple of different programming languages or editors or anything else for size. You may not end up choosing it, but seeing the possibilities and different ways of thinking is a very good thing.
sublinear 1y ago
May I humbly suggest that those files probably belong in an LFS submodule called "assets" or "vendor"? Then you can clone without checking out all the unnecessary large files to get a working build, This also helps on the legal side to correctly license your repos. I'm struggling to see how this is a problem with git and not just antipatterns that arise from badly organized projects.
- charcircuit 1y ago
  The user shouldn't have to think about such a thing. Version control should handle everything automatically and not force the user into doing extra work to workaround issues.
  - hinkley 1y ago
    I always hated the “write your code like the next maintainer is a psychopath” mantra because it makes the goal unclear. I prefer the following: Write your code/tools as if they will be used at 2:00 am while the server room is on fire. Because sooner or later they will be. A lot of our processes are used like emergency procedures. Emergency procedures are meant to be brainless as much as possible. So you can reserve the rest of your capacity for the actual problem. My version essentially calls out Kernighan’s Law.
  - sublinear 1y ago
    Organizing your files sensibly is not necessary to use LFS nor is it a "workaround". It's just a pattern I am suggesting to make life easier regardless of what tools you decide to use. I can't think of a case where organizing your project to fail gracefully is a bad idea. Git does the responsible thing and lets the user determine how to proceed with the mess they've made. I must say I'm increasingly suspicious of the hate that git receives these days.
  - yencabulator 1y ago
    Nothing ever handles everything automatically. Trade-offs always exist. Limiting scope is a sign of a healthy architecture.
- 1y ago
jameshart 1y ago
Nit: > if I git clone a repo with many revisions of a noisome 25 MB PNG file FYI ‘noisome’ is not a synonym for ‘noisy’ - it’s more of a synonym for ‘noxious’; it means something smells bad.
- deleted 1y ago
  [deleted]
- williadc 1y ago
  I believe that was the author's intent.
  - jameshart 1y ago
    I guess maybe it’s the nonstandard sMEL chunk that bumps the size of the PNG file up so high. Seemed more to me that they were talking about an image of random noise though.
- deleted 1y ago
  [deleted]
jiggawatts 1y ago
What I would love to see in an SCM that properly supports large binary blobs is storing the contents using Prolly trees instead of a simple SHA hash. Prolly trees are very similar to Merkle trees or the rsync algorithm, but they support mutation and version history retention with some nice properties. For example: you always obtain exactly the same tree (with the same root hash) irrespective of the order of incremental edit operations used to get to the same state. In other words, two users could edit a subset of a 1 TB file, both could merge their edits, and both will then agree on the root hash without having to re-hash or even download the entire file! Another major advantage on modern many-core CPUs is that Prolly trees can be constructed in parallel instead of having to be streamed sequentially on one thread. Then the really big brained move is to store the entire SCM repo as a single Prolly tree for efficient incremental downloads, merges, or whatever. I.e.: a repo fork could share storage with the original not just up to the point-in-time of the fork, but all future changes too.
- hinkley 1y ago
  Git has had a good run. Maybe it’s time for a new system built by someone who learned about DX early in their career, instead of via their own bug database. If there’s a new algorithm out there that warrants a look…
  - viraptor 1y ago
    Jujutsu unfortunately doesn't have any story for large files yet (as far as I can tell), but maybe soon ...
    - martinvonz 1y ago
      That's correct. It's only on our roadmap so far (https://jj-vcs.github.io/jj/latest/roadmap/#better-support-for-large-files https://jj-vcs.github.io/jj/latest/roadmap/#better-support-f...). We have also talked about doing something similar for tree objects in order to better support very large directories (to reduce the amount of data we need to transfer for them) and very deep directories (to reduce the number of roundtrips to the server). I think we have only talked about that on Discord so far (https://discord.com/channels/968932220549103686/969291218347524238/1295896123998732329 https://discord.com/channels/968932220549103686/969291218347...). It would not be compatible with Git repos, so it would only really be useful to teams outside Google once there's an external jj-native forge that decides to support it (if our rough design is even realistic).
bahmboo 1y ago
I'm just dipping my toe into Data Version Control - DVC. It is aimed towards data science and large digital asset management using configurable storage sources under a git meta layer. The goal is separation of concerns: git is used for versioning and the storage layers are dumb storage. Does anyone have feedback about personally using DVC vs LFS?
- bokchoi 1y ago
  It sounds like git-annex might be a good option for you. There is also https://www.datalad.org/ https://www.datalad.org/ built on top of git-annex for large data management.
  - mdaniel 1y ago
    Writing type-free Python in 2025 is malpractice
    - Hackbraten 1y ago
      To be fair, they use types for the complex parts. [0] [0]: https://github.com/datalad/datalad/blob/maint/datalad/support/gitrepo.py https://github.com/datalad/datalad/blob/maint/datalad/suppor...
- Evidlo 1y ago
  I did a simple test tracking a few hundred gigs of random /dev/urandom data. LFS choked on upload speed while DVC worked fine. My team is using DVC now
- memmel 1y ago
  I'm in the same boat - I decided this week for DVC over LFS. For me, the deciding factor was that with LFS, if you want to delete objects from storage, you have to rewrite git history. At least, that's what both the Github and Gitlab docs specify. DVC adds a layer of indirection, so that its structure is not directly tied to git. If I change my mind and delete the objects from S3, dvc might stop working, but git will be fine. Some extra pluses about DVC: - It can point to versioned S3 objects that you might already have as part of existing data pipelines. - It integrates with the Python fsspec library to read the files on demand using paths like "dvc://path/to/file.parquet". This feels nicer than needing to download all the files up front.
HexDecOctBin 1y ago
So this filter argument will reduce the repo size when cloning, but how will one reduce the repo size after a long stint of local commits of changing binary assets? Delete the repo and clone again?
- reactordev 1y ago
  yeah, this isn't really solving the problem. It's just punting it. While I welcome a short-circuit filter, I see dragons ahead. Dependencies. Assets. Models... won't benefit at all as these repos need the large files - hence why there are large files.
  - rezonant 1y ago
    There seems to be a misunderstanding. The --filter option simple doesn't populate content in the .git directory which is not required for the checkout. If there is a file that is large which is needed for the current checkout (ie the parts not in the .git folder), it will be fetched regardless of the filter option. To put it another way, regardless of what max size you give to --filter, you will end up with a complete git checkout, no missing files.
  - actuallyalys 1y ago
    It’s definitely not a full solution, but it seems like it would solve cases where having the full history of the large files available, just not on everyone’s machine, is the desired behavior.
- viraptor 1y ago
  It's really not clear which behaviour you want though. For example when you do lots of bisects you probably want to keep everything downloaded locally. If you're just working on new things, you may want to prune the old blobs. This information only exists in your head though.
  - HexDecOctBin 1y ago
    The ideal behaviour is so have a filter on push too, meaning that files above a certain size should be deleted from non-latest history after push.
nixpulvis 1y ago
I was just using git LFS and was very concerned with how bad the help message was compared to the rest of git. I know it seems small, but it just never felt like a team player, and now I'm very happy to hear this.
technoweenie 1y ago
I'm really happy to see large file support in Git core. Any external solution would have similar opt-in procedures. I really wanted it to work seamlessly with as few extra commands as possible, so the API was constrained to the smudge and clean filters in the '.gitattributes' file. Though I did work hard to remove any vendor lock-in by working directly with Atlassian and Microsoft pretty early in the process. It was a great working relationship, with a lot of help from Atlassian in particular on the file locking API. LFS shipped open source with compatible support in 3 separate git hosts.
firesteelrain 1y ago
We had a repo that was at one point 25GB. It had Git LFS turned on but the files weren’t stored outside of BitBucket. Whenever a build was run in Bamboo, it would choke big time. We found that we could move the large files to Artifactory as it has Git LFS support. But the problem was the entire history that did not have Artifactory pointers. Every clone included the large files (for some reason the filter functionality wouldn’t work for us - it was a large repo and it it had hundreds of users amongst other problems) Anyways what we ended up doing was closing that repo and opening a new one with the large files stripped. Nitpick in the authors page: “ Nowadays, there’s a free tier, but you’re dependent on the whims of GitHub to set pricing. Today, a 50GB repo on GitHub will cost $40/year for storage” This is not true as you don’t need GitHub to get LFS support
bob1029 1y ago
> Large object promisors are special Git remotes that only house large files. I like this approach. If I could configure my repos to use something like S3, I would switch away from using LFS. S3 seems like a really good synergy for large blobs in a VCS. The intelligent tiering feature can move data into colder tiers of storage as history naturally accumulates and old things are forgotten. I wouldn't mind a historical checkout taking half a day (i.e., restored from a robotic tape library) if I am pulling in stuff from a decade ago.
- account42 1y ago
  Why does the git client need specific support for this though? What's stopping a git host from redirecting requests certain objects to a different host and refusing to pack them into bundles today?
- deleted 1y ago
  [deleted]
- a_t48 1y ago
  At my current job I've started caching all of our LFS objects in a bucket, for cost reasons. Every time a PR is run, I get the list of objects via `git lfs ls-files`, sync them from gcp, run `git lfs checkout` to actually populate the repo from the object store, and then `git lfs pull` to pick up anything not cached. If there were uncached objects, I push them back up via `gcloud storage rsync`. Simple, doesn't require any configuration for developers (who only ever have to pull new objects), keeps the Github UI unconfused about the state of the repo. I'd initially at spinning up an LFS backends, but this solves the main pain point, for now. Github was charging us an arm and a leg for pulling LFS files for CI, because each checkout is fresh, the caching model is non-ideal (max 10GB cache, impossible to share between branches), so we end up pulling a bunch of data that is unfortunately in LFS, every commit, possibly multiple times. Because of this they happily charge us for all that bandwidth, because they don't provide tools to make it easy to reduce bandwidth (let me pay for more cache size, or warm workers with an entire cache disc, or better cache control, or...). ...and if I want to enable this for developers it's relatively easy, just add a new git hook to do the same set of operations locally.
jayd16 1y ago
How about git just fixes shallow clones and partial clones? Then we don't need convoluted work arounds to cheat in large content after we fully clone a history of pointers or promises or whatever. You should be able to set default clone depth by file type and size in the git attributes (or maybe a file that can also live above a repo like supporting attributes in .gitconfig locations?). Then the usual settings would be to shallow clone the latest content as well as fetch the full history and maybe the text file historical content. Ideally you could prune to the clone depth settings as well. Why are we still talking about large file pointers? If you fix shallow and partial clones, then any repo can be an efficient file mirror, right?
a_t48 1y ago
The real GH LFS cost is not the storage but the bandwidth on pulling objects down for every fresh clone. $$$$$. See my other comment. :)
- Imustaskforhelp 1y ago
  xet on hugging face does seem to not have such bandwidth issues imo. I wish that something like xet but open source could exist.
  - rajatarya 1y ago
    xet is open source, check out https://github.com/huggingface/xet-core https://github.com/huggingface/xet-core. We (I'm on the xet team at HF) are open sourcing our spec + protocol in the coming months. So, with the spec and protocol open-sourced, anyone can create xet clients and implement the protocol to build a xet backend. The specific implementation of our xet backend is deeply integrated into HF backend so open-sourcing it directly wouldn't be very helpful. Once we get the spec + protocol released it should be easy to generate a compatible backend.
    - Imustaskforhelp 1y ago
      Hm alright, so in the meanwhile, the best place to use xet might be huggingface I am wondering since a lot of people have said that for things like gamedev etc., git isn't best because of lfs issues and xet fixes it, could I use huggingface as a repository if I am gamedev or would that go against since huggingface is for AI stuff? Also thanks for open sourcing it, I hope that it would be in permissive license and open sourcing the spec + protocol might actually push the git community to be better which is really nice!
kerneltime 1y ago
https://github.com/oneconcern/datamon https://github.com/oneconcern/datamon Had written this git for data tool few years back (works with GCS but can be made to work with S3) 1. No server side 2. Immutable data (via GCS policies) 3. Ability to mount data sets as filesystems 4. Integrated with k8s. It was built to work for the needs of the startup funding it, but I would love it if it could be extended.
whatever1 1y ago
It is insane that almost after a century of running computations with data on computers we still don't have a good version control system that maps a code version to its relevant data version. Still the approach is to put code and data in a folder and call it a day. Slap a "_FINAL" at the folder name and you are golden.
- deleted 1y ago
  [deleted]
Ferret7446 1y ago
This article treats LFS unfairly. It does not in any way lock you in to GitHub; the protocol is open. The downsides of LFS are unavoidable as a Git extension. Promisors are basically the same concept as LFS, except as it's built into Git it is able to provide a better UX than is possible as an extension.
- andrewmcwatters 1y ago
  Using LFS once in a repository locks you in permanently. You actually have to delete the repository from GitHub to remove the space consumed. It’s entirely a non-starter. Nowhere is this behavior explicitly stated. I used to use Git LFS on GitHub to do my company’s study on GitHub statistics because we stored large compressed databases on users and repositories.
  - throwaway290 1y ago
    This conflates Git and Github. Github is crap, news at 11. Git itself is fine and LFS is an extension for Git. There is nothing in LFS spec that discusses storage billing. Anyone can write a better server
    - andrewmcwatters 1y ago
      It does because overwhelmingly the usage of Git is through GitHub. Everything else is practically a rounding error. So it’s incredibly helpful to know that the most popular large file retrieval extension to Git on the most popular Git host will lock you in.
      - throwaway290 1y ago
        That's very bad logic. By that logic Git sucks because Github sucks. I cringe every time people conflate it and if you know better then why... Just say Github sucks instead.
      - deleted 1y ago
        [deleted]
deleted 1y ago
[deleted]
anon-3988 1y ago
What prevents Git from simply working better with large files?
- AceJohnny2 1y ago
  git works just fine with large files. The problem is that when you clone a repo, or pull, by default it gets everything, including large files deep in the history that you probably don't care about anymore. That was actually an initial selling point of git: you have the full history locally. You can work from the plane/train/deserted island just fine. These large files will persist in the repo forever. So people look for options to segregate large files out so that they only get downloaded on demand (aka "lazily"). All the existing options (submodules, LFS, partial clones) are different answers to "how do we make certain files only download on demand"
  - anon-3988 1y ago
    IIRC, it take ages for it to index a large folder. I was trying to use it to store the diff of my backup folder that constantly get rclone'd and rsync'd over in case those fucked up catastrophically
  - nomel 1y ago
    > All the existing options Don't forget sparse checkouts!
  - expenses3 1y ago
    No, git does not work 'just fine' with large files. It works like ass.
deleted 1y ago
[deleted]
mathi0750 1y ago
Have you tried Oxen.ai? they are doing more fine-tuning and inference now but they have an open-source data version control platform written in rust at the core of their product.
gschoeni 1y ago
We're working on `oxen` to solve a lot of the problems we ran into with git or git-lfs. We have an open source CLI and server that mirrors git, but handles large files and mono repos with millions of files in a much more performant manner. Would love feedback if you want to check it out! https://github.com/Oxen-AI/Oxen https://github.com/Oxen-AI/Oxen
captn3m0 1y ago
Partial clones are also dependent on the server side supporting this. GitHub is one of the very few that does. git.kernel.org for eg did not, last I checked.
jeffWrld 1y ago
[dead]
wbillingsley 1y ago
What I used to recommend to my sofware engineering classes is that instead of putting large files (media etc) into Git, put them into the artifact repository (Artifactory or something like it). That lets you for instance publish it as a snapshot dependency that the build system will automatically fetch for you, but control how much history of it you keep and only require your colleagues to fetch the latest version. Even better, a simple clean of their build system cache will free up the space used by old versions on their machines.
- firesteelrain 1y ago
  Do you teach CI/CD systems architecture in your classes? Because I am finding that is what the junior engineers that we have hired seem to be missing. Tying it all in with GitLab, Artifactory, CodeSonar, Anchore etc
  - TZubiri 1y ago
    I think the OP refers to assets that truly belong in Git because they are source code but large, like 3d models. Release artifacts like a .exe would NOT belong in Git because it is not source code.
    - firesteelrain 1y ago
      I get it, I understood. I was forking the conversation for a sec
  - wbillingsley 1y ago
    Yes
- cyberax 1y ago
  This has its own issues. Now you need to provision additional credentials into your CI/CD and to your developers. Commits become multi-step, as you need to first commit the artifacts to get their artifact IDs to put in the repo. You can automate that via git hooks, but then you're back at where you started: git-lfs.
CSMastermind 1y ago
I could really use an alternative to Plastic SCM for 3D models. S3 is fine but lacks the nicities.
moonlion_eth 1y ago
the future of git is jj
- expenses3 1y ago
  Does it handle large files better? I thought it was just an improved interface over the git store.
ghhv 1y ago
[flagged]
pabs3 1y ago
The git storage model also needs an overhaul to be more like modern backup tools such as restic/borg; it needs content-defined chunking of files and directories.
ideasman42 1y ago
Something they missed out with the GitLFS cons is authentication, if you don't use SSH-agent, pushing involves authenticating multiple times, sometimes more than 2-3 times in my experience.
jokoon 1y ago
Git lfs seems controversial in the comments here Developers have their own drama
flohofwoe 1y ago
Even ancient svn works much better out of the box for large binary files than git (e.g. a 150 GB working directory with many big Photoshop files and other binary assets is no problem in SVN). What does SVN do differently than git when it comes to large binary files, and why can't git use the same approach? I also don't quite understand tbh how offloading large files to somewhere else would be fundamentally different than storing all files in one place except complicating everything? Storage is storage, how would a different storage location fix any of the current performance and robustness problems? Offloading just sounds like a solution for public git forges which don't want to deal with big files because it's too costly for them, but increased hosting cost is not the 'large binary file problem' of git. (edit: apparently git supports proper locking(?) so I removed that section - ps: nvm it looks like the file locking feature is only in git-lfs)
- pjc50 1y ago
  Completely different design. Git is intended to be fully distributed, so (a) every repo is supposed to have the full history of every file, and (b) locking is meaningless. People should use the VCS that's appropriate for their project rather than insist on git everywhere.
  - cyphar 1y ago
    To be clear, "fully distributed" also means "can use all of the features offline, and without needing commit access to the central repository". I can't imagine living without that feature, but I also do a lot of OSS work so I'm probably biased.
    - flohofwoe 1y ago
      How often are you fully offline though? A centralized version system could be updated to work in 'offline mode' by queueing pushed changes locally until internet is available again (and in SVN that would be quite trivial because it keeps the last known state of the repository in a local shadow copy anyway).
KronisLV 1y ago
> And the problems are significant: > High vendor lock-in – When GitHub wrote Git LFS, the other large file systems—Git Fat, Git Annex, and Git Media—were agnostic about the server-side. But GitHub locked users to their proprietary server implementation and charged folks to use it. Is this a current issue? I used Git LFS with a GitLab instance this week, seemed to work fine. https://docs.gitlab.com/topics/git/lfs/ https://docs.gitlab.com/topics/git/lfs/ I also used Git LFS with my Gitea instance a week before that, it was fine too. https://docs.gitea.com/administration/git-lfs-setup https://docs.gitea.com/administration/git-lfs-setup At the same time it feels odd to hear mentions of LFS being deprecated in the future, while I’ve seldom seen anyone even use it - people just don’t seem to care and shove images and such into regular Git which puzzles me.
rswail 1y ago
Is the problem that we don't have good "diff" equivalents for binaries that git could use to only store those diffs like the old RCS/CVS for large files?
- procaryote 1y ago
  subversion used to do that, actually probably still does... and also only checks out the latest revision. Svn is a bother in other ways of course, like being worse at regular version control, and only usable with access to the server etc. There's a bunch of binary files that change a lot on small changes due to compression or how the data is serialised, so the problem doesn't go away completely. One could conceivably start handling that, but there are lots of file formats out there and the sum oc complexity tends to be bugs and security issues.
  - rswail 1y ago
    Potentially with a new blob type but maintaining a reverse diff would be difficult as it would change the hash of the previous version if you had to store the diff. Another alternative would be storing the chunks as blobs so that you reconstruct the full binary and only have to store the changed chunks. However that doesn't work with compressed binaries.
- IshKebab 1y ago
  Not really. Git does use delta-based storage for binary files. It might not be as good as it could be for some files (e.g. compressed ones) but that's relatively easy to solve. The real problem is that Git wants you to have a full copy of all files that have ever existed in the repo. As soon as you add a large file to a repo it's there forever and can basically never be removed. If you keep editing it you'll build up lots more permanent data in the repo. Git is really missing: 1. A way to delete old data. 2. A way for the repo to indicate which data is probably not needed (old large binaries). 3. A way to serve large files efficiently (from a CDN). Some of these can sort of be done, but it's super janky. You have to proactively add confusing flags etc.
  - 1y ago
astrolx 1y ago
For what it's worth, I have a self hosted git forge where I upped the maximum filesize limit. I use git repositories for my science projects, they can each have hundreds of large files (> 25 Mb as described in the article). I don't use LFS, I don't encounter any issue.
tionis 1y ago
I've been using git + git-lfs to manage, sync and backup all my files (including media files and more) and it's quite convenient, but native support for large files would be great. I'd for example really like to be able to push large objects directly from one device to the next. At the moment I'm using my own git server + git lfs deduplication using btrfs to efficiently handle the large files. If large objects are just embedded in various packfiles this approach would no longer work, so I hope that such a behaviour can be controlled.
theli0nheart 1y ago
I wrote git-bigstore [0] almost 10 (!) years ago to solve this problem—even before Git LFS—and as far as I know, bigstore still works perfectly. You specify the files you want to store in your storage backend via .gitattributes, and use two separate commands to sync files. I have not touched this code in years but the general implementation should still work. GitHub launched LFS not too long after I wrote this, so I kind of gave up on the idea thinking that no one would want to use it in lieu of GitHub's solution, but based on the comments I think there's a place for it. It needs some love but the idea is solid. I wrote a little description on the wiki about the low-level implementation if you want to check it out. [1] Also, all of the metadata is stored using git notes, so is completely portable and is frontend agnostic—doesn't lock you into anything (except, of course, the storage backend you use). [0]: https://github.com/lionheart/git-bigstore https://github.com/lionheart/git-bigstore [1]: https://github.com/lionheart/git-bigstore/wiki https://github.com/lionheart/git-bigstore/wiki
- TiredOfLife 1y ago
  Earliest tagged git bigstore: Apr 26, 2016 Earliest tagged git lfs: Jun 25, 2014
  - theli0nheart 1y ago
    The first bigstore release on PyPi (0.1) was on March 27, 2013. Check this out: https://pypi.org/project/git-bigstore/#history https://pypi.org/project/git-bigstore/#history
- andrewmcwatters 1y ago
  You should... definitely not stop working on this. git-lfs is not good, and git-annex is far more awkward than it needs to be.
semperMade 1y ago
[dead]
Sophira 1y ago
How does this upcoming feature deal with the potential problem of fake commit IDs? Commit IDs are based on a number of factors about the commit, including the actual contents and the commit ID of the parent commit. Any fully cloned git repository can theoretically be audited to make sure that all its commit IDs are correct. Nobody does this (although perhaps git does automatically?), but it's possible. But now, picture a git repository that has a one petabyte file in one of its early commits (and deleted again later). Pretty much nobody is going to have the space required to download this, so many people will not even bother to do so. As such, what's to stop the server from just claiming any commit ID it wanted for this particular commit? Who's going to check? (Bonus: For that matter, is the one petabyte file even real? Or just a faked size in the metadata?) To be clear, I assume people have already thought about these issues. I'm just curious what the answers are.
- kbolino 1y ago
  If you think of a commit as a Merkle tree, then a file's content is a leaf node, and thus has nothing to verify. It either exists or it doesn't. Non-existence creates usability problems, but not verifiability problems. Of course, remote files can be used to sneak things in, but those things still have to get approved the same as any other commit content. You should not approve PRs etc. that reference remote files you haven't verified. And, while remote storage could be vulnerable to collision attacks in a way that git itself mostly isn't, git-lfs for example already uses SHA-256.
tpoacher 1y ago
That or rediscover the beauty of svn (only half-trolling)
yannick1976 1y ago
> large object promisors will work like this: - You push a large file to your Git host. - In the background, your Git host offloads that large file to a large object promisor. - When you clone, the Git host tells your Git client about the promisor. - Your client will clone from the Git host, and automagically nab large files from the promisor remote. This doesn’t seem to solve the issue of the long download time / big repo size. Or is the author describing the use of promisors remotes in conjunction with partial cloning? Unless I missed something it is not stated explicitly.
- krupan 1y ago
  Yes, in conjunction with shallow and/or partial cloning. The clone --filter is for shallow cloning

557 points by thcipriani 1y ago

As it should be! If it's not native to git, it's not worth using. I'm glad these issues are finally being solved. These new features are pretty awesome too. Especially separate large object remotes. They will probably enable git to be used for even more things than it's already being used for. They will enable new ways to work with git.

Affric 1y ago

Incredible. Nice to see some Microsoft and Google emails contributing.

Maxious 1y ago
Microsoft uses git for office development https://danielsada.tech/blog/carreer-part-7-how-office-moved-to-git-and-i-loved-devex/ https://danielsada.tech/blog/carreer-part-7-how-office-moved... Google has android and chromium as well as Git+Gerrit-on-Borg https://opensource.google/documentation/reference/releasing/hosting/ https://opensource.google/documentation/reference/releasing/...

als0 1y ago

10 years late is better than never.

tombert 1y ago

Is Git ever going to get proper support for binary files? I’ve never used it for anything serious but my understanding is that Mercurial handles binary files better? Like it supports binary diffs if I understand correctly. Any reason Git couldn’t get that?

ks2048 1y ago
I'm not sure binary diffs are the problem - e.g. for storing images or MP3s, binary diffs are usually worse than nothing.
- digikata 1y ago
  I would think that git would need a parallel storage scheme for binaries. Something that does binary chunking and deduplication between revisions, but keeps the same merkle referencing scheme as everything else.
  - tempay 1y ago
    > binary chunking and deduplication Are there many binaries that people would store in git where this would actually help? I assume most files end up with compression or some other form of randomization between revisions making deduplication futile.
    - hinkley 1y ago
      It would likely require more tooling.
    - digikata 1y ago
      I don't know, it's all probability in the dataset that makes one optimization strategy better over another. Git annex iirc does file level dedupe. That would take care of most of the problem if you're storing binaries that are compressed or encrypted. It's a lot of work to go beyond that, and probably one reason no one has bothered with git yet. But borg and restic both do chunked dedupe I think.
    - adastra22 1y ago
      A lot in the game and visual art industries.
    - zigzag312 1y ago
      2-3x reduction in repository size compared to Git LFS in this test: https://xethub.com/blog/benchmarking-the-modern-development-experience https://xethub.com/blog/benchmarking-the-modern-development-...
  - zigzag312 1y ago
    Xet uses block level deduplication.
firesteelrain 1y ago
A lot of people use Perforce Helix and others use Plastic SCM. That’s been my experience for like large binary assets with git-like functionality
- tom_ 1y ago
  I didn't enjoy using Plastic, but Perforce is ok (not to say that it's perfect - I miss a lot of git stuff). It does have no problems with lots of data though! This article moans about the overhead of a 25 MB png file... it's been a long time since i worked on a serious project where the head revision is less than 25 GB. Typical daily churn would be 2.5 GB+. (It's been even longer since i used svn in anger, but maybe it could work too. It has file locking, and local storage cost is proportional to size of head revision. It was manageable enough with a 2 GB head revision. Metadata access speed was always terrible though, which was tedious.)
  - firesteelrain 1y ago
    SVN should be able to handle large files no issue imho
brucehoult 1y ago
All files in git are binary files. All deltas between versions are binary diffs. Git has always handled large (including large binary) files just fine. What it doesn't like is files where a conceptually minor change changes the entire file, for example compressed or encrypted files. The only somewhat valid complaint is that if someone once committed a large file and then it was later deleted (maybe minutes later, maybe years later) then it is in the repo and in everyone's checkouts forever. Which applies equally to small and to large files, but large ones have more impact. That's the whole point of a version control system. To preserve the history, allowing earlier versions to be recreated. The better solution would be to have better review of changes pushed to the master repo, including having unreviewed changes in separate, potentially sacrificial, repos until approved.
afiori 1y ago
My understanding is that git diff algorithms require a file to be segmentable (eg text files are split line-wise) and there is no general segmentation strategy for binary blobs. But a good segmentation is only good for better compression and nicer diff, git could do byte wise diffs with no issues, so I wonder why doesn't git use customizable segmentation strategies where it calls external tools based on file type (eg a rust thingy for rust file etc, or a PNG thingy for PNG files). At worst the tool would return either a single segment for the entire file or the byte wise split which would work anyway
- joeyh 1y ago
  A common misconception. git has always used binary deltas for pack files. Consider that git tree objects are themselves not text files, and git needs efficiently store slightly modified versions of the same tree.
  - afiori 1y ago
    so are pack files diffed byte-wise?

jauer 1y ago

TFA asserts that Git LFS is bad for several reasons including because proprietary with vendor lock-in which I don't think is fair to claim. GitHub provided an open client and server which negates that. LFS does break disconnected/offline/sneakernet operations which wasn't mentioned and is not awesome, but those are niche workflows. It sounds like that would also be broken with promisors. The `git partial clone` examples are cool! The description of Large Object Promisors makes it sound like they take the client-side complexity in LFS, move it server-side, and then increases the complexity? Instead of the client uploading to a git server and to a LFS server it uploads to a git server which in turn uploads to an object store, but the client will download directly from the object store? Obviously different tradeoffs there. I'm curious how often people will get bit by uploading to public git servers which upload to hidden promisor remotes.

IshKebab 1y ago
LFS is bad. The server implementations suck. It conflates object contents with the storage method. It's opt-in, in a terrible way - if you do the obvious thing you get tiny text files instead of the files you actually want. I dunno if their solution is any better but it's fairly unarguable that LFS is bad.
- jayd16 1y ago
  It does seem like this proposal has exactly the same issue. Unless this new method blocks cloning when unable to access the promisors, you'll end up with similar problems of broken large files.
  - cowsandmilk 1y ago
    How so? This proposal doesn’t require you to run `git lfs install` to get the correct files…
    - jayd16 1y ago
      If the architecture is irrelevant and it's just a matter of turning it on by default they could have done that with LFS long ago.

glitchc 1y ago

No. This is not a solution. While git LFS is just a kludge for now, writing a filter argument during the clone operation is not the long-term solution either. Git clone is the very first command most people will run when learning how to use git. Emphasized for effect: the very first command. Will they remember to write the filter? Maybe, if the tutorial to the cool codebase they're trying to access mentions it. Maybe not. What happens if they don't? It may take a long time without any obvious indication. And if they do? The cloned repo might not be compilable/usable since the blobs are missing. Say they do get it right. Will they understand it? Most likely not. We are exposing the inner workings of git on the very first command they learn. What's a blob? Why do I need to filter on it? Where are blobs stored? It's classic abstraction leakage. This is a solved problem: Rsync does it. Just port the bloody implementation over. It does mean supporting alternative representations or moving away from blobs altogether, which git maintainers seem unwilling to do.

IshKebab 1y ago
I totally agree. This follows a long tradition of Git "fixing" things by adding a flag that 99% of users won't ever discover. They never fix the defaults. And yes, you can fix defaults without breaking backwards compatibility.
- Jenk 1y ago
  > They never fix the defaults Not strictly true. They did change the default push behaviour from "matching" to "simple" in Git 2.0.
  - hinkley 1y ago
    So what was the second time the stopped watch was right? I agree with GP. The git community is very fond of doing checkbox fixes for team problems that aren’t or can’t be set as defaults and so require constant user intervention to work. See also some of the sparse checkout systems and adding notes to commits after the fact. They only work if you turn every pull and push into a flurry of activity. Which means they will never work from your IDE. Those are non fixes that pollute the space for actual fixes.

goneri 1y ago

git-annex is a good alternative to the solution of Githu, and it supports different storage backends. I'm actually surprised it's not more popular.

deleted 1y ago
[deleted]
alchemist1e9 1y ago
Was looking for a mention of git-annex and I completely agree. I’ve used it extensively and have found it works really well. Any ideas why it isn’t more popular and more well known?
- cesarb 1y ago
  > Any ideas why it isn’t more popular and more well known? While git-annex works very well on Unix-style systems with Unix-style filesystems, it heavily depends on symbolic links, which do not exist on filesystems like exFAT, and are problematic on Windows (AFAIK, you have to be an administrator, or enable an obscure group policy). It has a degraded mode for these filesystems, but uses twice the disk space in that mode, and AFAIK loses some features.
- avar 1y ago
  I use it, and love it. But it's not intended for or good at (without forcing a square peg into a round hole) the sort of thing LFS and promisors are for, which is a public project with binary assets. git-annex is really for (and shines at) a private backup solution where you'd like to have N copies of some data around on various storage devices, track the history of each copy, ensure that you have at least N copies etc. Each repository gets a UUID, and each tracked file has a SHA-256 hash. There's a branch which has a timestamp and repo UUID to SHA-256 mapping, if you have 10 repos that file will have (at least) 10 entries. You can "trust" different repositories to different degrees, e.g. if you're storing a file on both some RAID'd storage server, or an old portable HD you're keeping in a desk drawer. This really doesn't scale for a public project. E.g. I have a repository that I back up my photos and videos in, that repository has ~700 commits, and ~6000 commits to the metadata "git-annex" branch, pretty close to a 1:10 ratio. There's an exhaustive history of every file movement that's ever occurred on the 10 storage devices I've ever used for that repository. Now imagine doing all that on a project used by more than one person. All other solutions to tracking large files along with a git repository forgo all this complexity in favor of basically saying "just get the rest where you cloned me from, they'll have it!".

forrestthewoods 1y ago

Git is fundamentally broken and bad. Almost all projects are defacto centralized. Your project is not Linux. A good version control system would support petabyte scale history and terabyte scale clones via sparse virtual filesystem. Git’s design is just bad for almost all projects that aren’t Linux. (I know this will get downvoted. But most modern programmers have never used anything but Git and so they don’t realize their tool is actually quite bad! It’s a shame.)

matheusmoreira 1y ago
Completely disagree. Git is fundamentally functional and good. All projects are local and decentralized, and any "centralization" is in fact just git hosting services, of which there are many options which are not even mutually exclusive.
- the_arun 1y ago
  Are you missing the central hosting services provide a good backup plan for your locally hosted git?
  - matheusmoreira 1y ago
    I agree! They are excellent git backup services. I use several of them: github, codeberg, gitlab, sourcehut. I can easily set up remotes to push to all of them at once. I also have copies of my important repositories on all my personal computers, including my phone. This is only possible because git is decentralized. Claiming that git is centralized is complete falsehood.
- compiler-guy 1y ago
  Got works fine and is solid and well enough known to be a reasonable choice for most people. But I encourage everyone to try out a few alternatives (and adopt their workflows at least for a while). I have no idea if you have or not. But fine has never used the alternatives, one doesn’t really know just how nice things can be. Or, even if you still find fit to be your preferred can, having an alternative experience can open you to other possibilities and ways of working. Just like everyone should try a couple of different programming languages or editors or anything else for size. You may not end up choosing it, but seeing the possibilities and different ways of thinking is a very good thing.

sublinear 1y ago

May I humbly suggest that those files probably belong in an LFS submodule called "assets" or "vendor"? Then you can clone without checking out all the unnecessary large files to get a working build, This also helps on the legal side to correctly license your repos. I'm struggling to see how this is a problem with git and not just antipatterns that arise from badly organized projects.

charcircuit 1y ago
The user shouldn't have to think about such a thing. Version control should handle everything automatically and not force the user into doing extra work to workaround issues.
- hinkley 1y ago
  I always hated the “write your code like the next maintainer is a psychopath” mantra because it makes the goal unclear. I prefer the following: Write your code/tools as if they will be used at 2:00 am while the server room is on fire. Because sooner or later they will be. A lot of our processes are used like emergency procedures. Emergency procedures are meant to be brainless as much as possible. So you can reserve the rest of your capacity for the actual problem. My version essentially calls out Kernighan’s Law.
- sublinear 1y ago
  Organizing your files sensibly is not necessary to use LFS nor is it a "workaround". It's just a pattern I am suggesting to make life easier regardless of what tools you decide to use. I can't think of a case where organizing your project to fail gracefully is a bad idea. Git does the responsible thing and lets the user determine how to proceed with the mess they've made. I must say I'm increasingly suspicious of the hate that git receives these days.
- yencabulator 1y ago
  Nothing ever handles everything automatically. Trade-offs always exist. Limiting scope is a sign of a healthy architecture.
1y ago

jameshart 1y ago

Nit: > if I git clone a repo with many revisions of a noisome 25 MB PNG file FYI ‘noisome’ is not a synonym for ‘noisy’ - it’s more of a synonym for ‘noxious’; it means something smells bad.

deleted 1y ago
[deleted]
williadc 1y ago
I believe that was the author's intent.
- jameshart 1y ago
  I guess maybe it’s the nonstandard sMEL chunk that bumps the size of the PNG file up so high. Seemed more to me that they were talking about an image of random noise though.
deleted 1y ago
[deleted]

jiggawatts 1y ago

What I would love to see in an SCM that properly supports large binary blobs is storing the contents using Prolly trees instead of a simple SHA hash. Prolly trees are very similar to Merkle trees or the rsync algorithm, but they support mutation and version history retention with some nice properties. For example: you always obtain exactly the same tree (with the same root hash) irrespective of the order of incremental edit operations used to get to the same state. In other words, two users could edit a subset of a 1 TB file, both could merge their edits, and both will then agree on the root hash without having to re-hash or even download the entire file! Another major advantage on modern many-core CPUs is that Prolly trees can be constructed in parallel instead of having to be streamed sequentially on one thread. Then the really big brained move is to store the entire SCM repo as a single Prolly tree for efficient incremental downloads, merges, or whatever. I.e.: a repo fork could share storage with the original not just up to the point-in-time of the fork, but all future changes too.

hinkley 1y ago
Git has had a good run. Maybe it’s time for a new system built by someone who learned about DX early in their career, instead of via their own bug database. If there’s a new algorithm out there that warrants a look…
- viraptor 1y ago
  Jujutsu unfortunately doesn't have any story for large files yet (as far as I can tell), but maybe soon ...
  - martinvonz 1y ago
    That's correct. It's only on our roadmap so far (https://jj-vcs.github.io/jj/latest/roadmap/#better-support-for-large-files https://jj-vcs.github.io/jj/latest/roadmap/#better-support-f...). We have also talked about doing something similar for tree objects in order to better support very large directories (to reduce the amount of data we need to transfer for them) and very deep directories (to reduce the number of roundtrips to the server). I think we have only talked about that on Discord so far (https://discord.com/channels/968932220549103686/969291218347524238/1295896123998732329 https://discord.com/channels/968932220549103686/969291218347...). It would not be compatible with Git repos, so it would only really be useful to teams outside Google once there's an external jj-native forge that decides to support it (if our rough design is even realistic).

bahmboo 1y ago

I'm just dipping my toe into Data Version Control - DVC. It is aimed towards data science and large digital asset management using configurable storage sources under a git meta layer. The goal is separation of concerns: git is used for versioning and the storage layers are dumb storage. Does anyone have feedback about personally using DVC vs LFS?

bokchoi 1y ago
It sounds like git-annex might be a good option for you. There is also https://www.datalad.org/ https://www.datalad.org/ built on top of git-annex for large data management.
- mdaniel 1y ago
  Writing type-free Python in 2025 is malpractice
  - Hackbraten 1y ago
    To be fair, they use types for the complex parts. [0] [0]: https://github.com/datalad/datalad/blob/maint/datalad/support/gitrepo.py https://github.com/datalad/datalad/blob/maint/datalad/suppor...
Evidlo 1y ago
I did a simple test tracking a few hundred gigs of random /dev/urandom data. LFS choked on upload speed while DVC worked fine. My team is using DVC now
memmel 1y ago
I'm in the same boat - I decided this week for DVC over LFS. For me, the deciding factor was that with LFS, if you want to delete objects from storage, you have to rewrite git history. At least, that's what both the Github and Gitlab docs specify. DVC adds a layer of indirection, so that its structure is not directly tied to git. If I change my mind and delete the objects from S3, dvc might stop working, but git will be fine. Some extra pluses about DVC: - It can point to versioned S3 objects that you might already have as part of existing data pipelines. - It integrates with the Python fsspec library to read the files on demand using paths like "dvc://path/to/file.parquet". This feels nicer than needing to download all the files up front.

HexDecOctBin 1y ago

So this filter argument will reduce the repo size when cloning, but how will one reduce the repo size after a long stint of local commits of changing binary assets? Delete the repo and clone again?

reactordev 1y ago
yeah, this isn't really solving the problem. It's just punting it. While I welcome a short-circuit filter, I see dragons ahead. Dependencies. Assets. Models... won't benefit at all as these repos need the large files - hence why there are large files.
- rezonant 1y ago
  There seems to be a misunderstanding. The --filter option simple doesn't populate content in the .git directory which is not required for the checkout. If there is a file that is large which is needed for the current checkout (ie the parts not in the .git folder), it will be fetched regardless of the filter option. To put it another way, regardless of what max size you give to --filter, you will end up with a complete git checkout, no missing files.
- actuallyalys 1y ago
  It’s definitely not a full solution, but it seems like it would solve cases where having the full history of the large files available, just not on everyone’s machine, is the desired behavior.
viraptor 1y ago
It's really not clear which behaviour you want though. For example when you do lots of bisects you probably want to keep everything downloaded locally. If you're just working on new things, you may want to prune the old blobs. This information only exists in your head though.
- HexDecOctBin 1y ago
  The ideal behaviour is so have a filter on push too, meaning that files above a certain size should be deleted from non-latest history after push.

nixpulvis 1y ago

I was just using git LFS and was very concerned with how bad the help message was compared to the rest of git. I know it seems small, but it just never felt like a team player, and now I'm very happy to hear this.

technoweenie 1y ago

I'm really happy to see large file support in Git core. Any external solution would have similar opt-in procedures. I really wanted it to work seamlessly with as few extra commands as possible, so the API was constrained to the smudge and clean filters in the '.gitattributes' file. Though I did work hard to remove any vendor lock-in by working directly with Atlassian and Microsoft pretty early in the process. It was a great working relationship, with a lot of help from Atlassian in particular on the file locking API. LFS shipped open source with compatible support in 3 separate git hosts.

firesteelrain 1y ago

We had a repo that was at one point 25GB. It had Git LFS turned on but the files weren’t stored outside of BitBucket. Whenever a build was run in Bamboo, it would choke big time. We found that we could move the large files to Artifactory as it has Git LFS support. But the problem was the entire history that did not have Artifactory pointers. Every clone included the large files (for some reason the filter functionality wouldn’t work for us - it was a large repo and it it had hundreds of users amongst other problems) Anyways what we ended up doing was closing that repo and opening a new one with the large files stripped. Nitpick in the authors page: “ Nowadays, there’s a free tier, but you’re dependent on the whims of GitHub to set pricing. Today, a 50GB repo on GitHub will cost $40/year for storage” This is not true as you don’t need GitHub to get LFS support

bob1029 1y ago

> Large object promisors are special Git remotes that only house large files. I like this approach. If I could configure my repos to use something like S3, I would switch away from using LFS. S3 seems like a really good synergy for large blobs in a VCS. The intelligent tiering feature can move data into colder tiers of storage as history naturally accumulates and old things are forgotten. I wouldn't mind a historical checkout taking half a day (i.e., restored from a robotic tape library) if I am pulling in stuff from a decade ago.

account42 1y ago
Why does the git client need specific support for this though? What's stopping a git host from redirecting requests certain objects to a different host and refusing to pack them into bundles today?
deleted 1y ago
[deleted]
a_t48 1y ago
At my current job I've started caching all of our LFS objects in a bucket, for cost reasons. Every time a PR is run, I get the list of objects via `git lfs ls-files`, sync them from gcp, run `git lfs checkout` to actually populate the repo from the object store, and then `git lfs pull` to pick up anything not cached. If there were uncached objects, I push them back up via `gcloud storage rsync`. Simple, doesn't require any configuration for developers (who only ever have to pull new objects), keeps the Github UI unconfused about the state of the repo. I'd initially at spinning up an LFS backends, but this solves the main pain point, for now. Github was charging us an arm and a leg for pulling LFS files for CI, because each checkout is fresh, the caching model is non-ideal (max 10GB cache, impossible to share between branches), so we end up pulling a bunch of data that is unfortunately in LFS, every commit, possibly multiple times. Because of this they happily charge us for all that bandwidth, because they don't provide tools to make it easy to reduce bandwidth (let me pay for more cache size, or warm workers with an entire cache disc, or better cache control, or...). ...and if I want to enable this for developers it's relatively easy, just add a new git hook to do the same set of operations locally.

jayd16 1y ago

How about git just fixes shallow clones and partial clones? Then we don't need convoluted work arounds to cheat in large content after we fully clone a history of pointers or promises or whatever. You should be able to set default clone depth by file type and size in the git attributes (or maybe a file that can also live above a repo like supporting attributes in .gitconfig locations?). Then the usual settings would be to shallow clone the latest content as well as fetch the full history and maybe the text file historical content. Ideally you could prune to the clone depth settings as well. Why are we still talking about large file pointers? If you fix shallow and partial clones, then any repo can be an efficient file mirror, right?

a_t48 1y ago

The real GH LFS cost is not the storage but the bandwidth on pulling objects down for every fresh clone. $$$$$. See my other comment. :)

Imustaskforhelp 1y ago
xet on hugging face does seem to not have such bandwidth issues imo. I wish that something like xet but open source could exist.
- rajatarya 1y ago
  xet is open source, check out https://github.com/huggingface/xet-core https://github.com/huggingface/xet-core. We (I'm on the xet team at HF) are open sourcing our spec + protocol in the coming months. So, with the spec and protocol open-sourced, anyone can create xet clients and implement the protocol to build a xet backend. The specific implementation of our xet backend is deeply integrated into HF backend so open-sourcing it directly wouldn't be very helpful. Once we get the spec + protocol released it should be easy to generate a compatible backend.
  - Imustaskforhelp 1y ago
    Hm alright, so in the meanwhile, the best place to use xet might be huggingface I am wondering since a lot of people have said that for things like gamedev etc., git isn't best because of lfs issues and xet fixes it, could I use huggingface as a repository if I am gamedev or would that go against since huggingface is for AI stuff? Also thanks for open sourcing it, I hope that it would be in permissive license and open sourcing the spec + protocol might actually push the git community to be better which is really nice!

kerneltime 1y ago

https://github.com/oneconcern/datamon https://github.com/oneconcern/datamon Had written this git for data tool few years back (works with GCS but can be made to work with S3) 1. No server side 2. Immutable data (via GCS policies) 3. Ability to mount data sets as filesystems 4. Integrated with k8s. It was built to work for the needs of the startup funding it, but I would love it if it could be extended.

whatever1 1y ago

It is insane that almost after a century of running computations with data on computers we still don't have a good version control system that maps a code version to its relevant data version. Still the approach is to put code and data in a folder and call it a day. Slap a "_FINAL" at the folder name and you are golden.

deleted 1y ago
[deleted]

Ferret7446 1y ago

This article treats LFS unfairly. It does not in any way lock you in to GitHub; the protocol is open. The downsides of LFS are unavoidable as a Git extension. Promisors are basically the same concept as LFS, except as it's built into Git it is able to provide a better UX than is possible as an extension.

andrewmcwatters 1y ago
Using LFS once in a repository locks you in permanently. You actually have to delete the repository from GitHub to remove the space consumed. It’s entirely a non-starter. Nowhere is this behavior explicitly stated. I used to use Git LFS on GitHub to do my company’s study on GitHub statistics because we stored large compressed databases on users and repositories.
- throwaway290 1y ago
  This conflates Git and Github. Github is crap, news at 11. Git itself is fine and LFS is an extension for Git. There is nothing in LFS spec that discusses storage billing. Anyone can write a better server
  - andrewmcwatters 1y ago
    It does because overwhelmingly the usage of Git is through GitHub. Everything else is practically a rounding error. So it’s incredibly helpful to know that the most popular large file retrieval extension to Git on the most popular Git host will lock you in.
    - throwaway290 1y ago
      That's very bad logic. By that logic Git sucks because Github sucks. I cringe every time people conflate it and if you know better then why... Just say Github sucks instead.
    - deleted 1y ago
      [deleted]

deleted 1y ago

[deleted]

anon-3988 1y ago

What prevents Git from simply working better with large files?

AceJohnny2 1y ago
git works just fine with large files. The problem is that when you clone a repo, or pull, by default it gets everything, including large files deep in the history that you probably don't care about anymore. That was actually an initial selling point of git: you have the full history locally. You can work from the plane/train/deserted island just fine. These large files will persist in the repo forever. So people look for options to segregate large files out so that they only get downloaded on demand (aka "lazily"). All the existing options (submodules, LFS, partial clones) are different answers to "how do we make certain files only download on demand"
- anon-3988 1y ago
  IIRC, it take ages for it to index a large folder. I was trying to use it to store the diff of my backup folder that constantly get rclone'd and rsync'd over in case those fucked up catastrophically
- nomel 1y ago
  > All the existing options Don't forget sparse checkouts!
- expenses3 1y ago
  No, git does not work 'just fine' with large files. It works like ass.

deleted 1y ago

[deleted]

mathi0750 1y ago

Have you tried Oxen.ai? they are doing more fine-tuning and inference now but they have an open-source data version control platform written in rust at the core of their product.

gschoeni 1y ago

We're working on `oxen` to solve a lot of the problems we ran into with git or git-lfs. We have an open source CLI and server that mirrors git, but handles large files and mono repos with millions of files in a much more performant manner. Would love feedback if you want to check it out! https://github.com/Oxen-AI/Oxen https://github.com/Oxen-AI/Oxen

captn3m0 1y ago

Partial clones are also dependent on the server side supporting this. GitHub is one of the very few that does. git.kernel.org for eg did not, last I checked.

jeffWrld 1y ago

[dead]

wbillingsley 1y ago

What I used to recommend to my sofware engineering classes is that instead of putting large files (media etc) into Git, put them into the artifact repository (Artifactory or something like it). That lets you for instance publish it as a snapshot dependency that the build system will automatically fetch for you, but control how much history of it you keep and only require your colleagues to fetch the latest version. Even better, a simple clean of their build system cache will free up the space used by old versions on their machines.

firesteelrain 1y ago
Do you teach CI/CD systems architecture in your classes? Because I am finding that is what the junior engineers that we have hired seem to be missing. Tying it all in with GitLab, Artifactory, CodeSonar, Anchore etc
- TZubiri 1y ago
  I think the OP refers to assets that truly belong in Git because they are source code but large, like 3d models. Release artifacts like a .exe would NOT belong in Git because it is not source code.
  - firesteelrain 1y ago
    I get it, I understood. I was forking the conversation for a sec
- wbillingsley 1y ago
  Yes
cyberax 1y ago
This has its own issues. Now you need to provision additional credentials into your CI/CD and to your developers. Commits become multi-step, as you need to first commit the artifacts to get their artifact IDs to put in the repo. You can automate that via git hooks, but then you're back at where you started: git-lfs.

CSMastermind 1y ago

I could really use an alternative to Plastic SCM for 3D models. S3 is fine but lacks the nicities.

moonlion_eth 1y ago

the future of git is jj

expenses3 1y ago
Does it handle large files better? I thought it was just an improved interface over the git store.

ghhv 1y ago

[flagged]

pabs3 1y ago

The git storage model also needs an overhaul to be more like modern backup tools such as restic/borg; it needs content-defined chunking of files and directories.

ideasman42 1y ago

Something they missed out with the GitLFS cons is authentication, if you don't use SSH-agent, pushing involves authenticating multiple times, sometimes more than 2-3 times in my experience.

jokoon 1y ago

Git lfs seems controversial in the comments here Developers have their own drama

flohofwoe 1y ago

Even ancient svn works much better out of the box for large binary files than git (e.g. a 150 GB working directory with many big Photoshop files and other binary assets is no problem in SVN). What does SVN do differently than git when it comes to large binary files, and why can't git use the same approach? I also don't quite understand tbh how offloading large files to somewhere else would be fundamentally different than storing all files in one place except complicating everything? Storage is storage, how would a different storage location fix any of the current performance and robustness problems? Offloading just sounds like a solution for public git forges which don't want to deal with big files because it's too costly for them, but increased hosting cost is not the 'large binary file problem' of git. (edit: apparently git supports proper locking(?) so I removed that section - ps: nvm it looks like the file locking feature is only in git-lfs)

pjc50 1y ago
Completely different design. Git is intended to be fully distributed, so (a) every repo is supposed to have the full history of every file, and (b) locking is meaningless. People should use the VCS that's appropriate for their project rather than insist on git everywhere.
- cyphar 1y ago
  To be clear, "fully distributed" also means "can use all of the features offline, and without needing commit access to the central repository". I can't imagine living without that feature, but I also do a lot of OSS work so I'm probably biased.
  - flohofwoe 1y ago
    How often are you fully offline though? A centralized version system could be updated to work in 'offline mode' by queueing pushed changes locally until internet is available again (and in SVN that would be quite trivial because it keeps the last known state of the repository in a local shadow copy anyway).

KronisLV 1y ago

> And the problems are significant: > High vendor lock-in – When GitHub wrote Git LFS, the other large file systems—Git Fat, Git Annex, and Git Media—were agnostic about the server-side. But GitHub locked users to their proprietary server implementation and charged folks to use it. Is this a current issue? I used Git LFS with a GitLab instance this week, seemed to work fine. https://docs.gitlab.com/topics/git/lfs/ https://docs.gitlab.com/topics/git/lfs/ I also used Git LFS with my Gitea instance a week before that, it was fine too. https://docs.gitea.com/administration/git-lfs-setup https://docs.gitea.com/administration/git-lfs-setup At the same time it feels odd to hear mentions of LFS being deprecated in the future, while I’ve seldom seen anyone even use it - people just don’t seem to care and shove images and such into regular Git which puzzles me.

rswail 1y ago

Is the problem that we don't have good "diff" equivalents for binaries that git could use to only store those diffs like the old RCS/CVS for large files?

procaryote 1y ago
subversion used to do that, actually probably still does... and also only checks out the latest revision. Svn is a bother in other ways of course, like being worse at regular version control, and only usable with access to the server etc. There's a bunch of binary files that change a lot on small changes due to compression or how the data is serialised, so the problem doesn't go away completely. One could conceivably start handling that, but there are lots of file formats out there and the sum oc complexity tends to be bugs and security issues.
- rswail 1y ago
  Potentially with a new blob type but maintaining a reverse diff would be difficult as it would change the hash of the previous version if you had to store the diff. Another alternative would be storing the chunks as blobs so that you reconstruct the full binary and only have to store the changed chunks. However that doesn't work with compressed binaries.
IshKebab 1y ago
Not really. Git does use delta-based storage for binary files. It might not be as good as it could be for some files (e.g. compressed ones) but that's relatively easy to solve. The real problem is that Git wants you to have a full copy of all files that have ever existed in the repo. As soon as you add a large file to a repo it's there forever and can basically never be removed. If you keep editing it you'll build up lots more permanent data in the repo. Git is really missing: 1. A way to delete old data. 2. A way for the repo to indicate which data is probably not needed (old large binaries). 3. A way to serve large files efficiently (from a CDN). Some of these can sort of be done, but it's super janky. You have to proactively add confusing flags etc.
- 1y ago

astrolx 1y ago

For what it's worth, I have a self hosted git forge where I upped the maximum filesize limit. I use git repositories for my science projects, they can each have hundreds of large files (> 25 Mb as described in the article). I don't use LFS, I don't encounter any issue.

tionis 1y ago

I've been using git + git-lfs to manage, sync and backup all my files (including media files and more) and it's quite convenient, but native support for large files would be great. I'd for example really like to be able to push large objects directly from one device to the next. At the moment I'm using my own git server + git lfs deduplication using btrfs to efficiently handle the large files. If large objects are just embedded in various packfiles this approach would no longer work, so I hope that such a behaviour can be controlled.

theli0nheart 1y ago

I wrote git-bigstore [0] almost 10 (!) years ago to solve this problem—even before Git LFS—and as far as I know, bigstore still works perfectly. You specify the files you want to store in your storage backend via .gitattributes, and use two separate commands to sync files. I have not touched this code in years but the general implementation should still work. GitHub launched LFS not too long after I wrote this, so I kind of gave up on the idea thinking that no one would want to use it in lieu of GitHub's solution, but based on the comments I think there's a place for it. It needs some love but the idea is solid. I wrote a little description on the wiki about the low-level implementation if you want to check it out. [1] Also, all of the metadata is stored using git notes, so is completely portable and is frontend agnostic—doesn't lock you into anything (except, of course, the storage backend you use). [0]: https://github.com/lionheart/git-bigstore https://github.com/lionheart/git-bigstore [1]: https://github.com/lionheart/git-bigstore/wiki https://github.com/lionheart/git-bigstore/wiki

TiredOfLife 1y ago
Earliest tagged git bigstore: Apr 26, 2016 Earliest tagged git lfs: Jun 25, 2014
- theli0nheart 1y ago
  The first bigstore release on PyPi (0.1) was on March 27, 2013. Check this out: https://pypi.org/project/git-bigstore/#history https://pypi.org/project/git-bigstore/#history
andrewmcwatters 1y ago
You should... definitely not stop working on this. git-lfs is not good, and git-annex is far more awkward than it needs to be.

semperMade 1y ago

[dead]

Sophira 1y ago

How does this upcoming feature deal with the potential problem of fake commit IDs? Commit IDs are based on a number of factors about the commit, including the actual contents and the commit ID of the parent commit. Any fully cloned git repository can theoretically be audited to make sure that all its commit IDs are correct. Nobody does this (although perhaps git does automatically?), but it's possible. But now, picture a git repository that has a one petabyte file in one of its early commits (and deleted again later). Pretty much nobody is going to have the space required to download this, so many people will not even bother to do so. As such, what's to stop the server from just claiming any commit ID it wanted for this particular commit? Who's going to check? (Bonus: For that matter, is the one petabyte file even real? Or just a faked size in the metadata?) To be clear, I assume people have already thought about these issues. I'm just curious what the answers are.

kbolino 1y ago
If you think of a commit as a Merkle tree, then a file's content is a leaf node, and thus has nothing to verify. It either exists or it doesn't. Non-existence creates usability problems, but not verifiability problems. Of course, remote files can be used to sneak things in, but those things still have to get approved the same as any other commit content. You should not approve PRs etc. that reference remote files you haven't verified. And, while remote storage could be vulnerable to collision attacks in a way that git itself mostly isn't, git-lfs for example already uses SHA-256.

tpoacher 1y ago

That or rediscover the beauty of svn (only half-trolling)

yannick1976 1y ago

> large object promisors will work like this: - You push a large file to your Git host. - In the background, your Git host offloads that large file to a large object promisor. - When you clone, the Git host tells your Git client about the promisor. - Your client will clone from the Git host, and automagically nab large files from the promisor remote. This doesn’t seem to solve the issue of the long download time / big repo size. Or is the author describing the use of promisors remotes in conjunction with partial cloning? Unless I missed something it is not stated explicitly.

krupan 1y ago
Yes, in conjunction with shallow and/or partial cloning. The clone --filter is for shallow cloning