6 ms·

One of the things that surprises me about Git is that it doesn’t store diffs. It just stores the full content of every file every time it’s modified and committ

One of the things that surprises me about Git is that it doesn’t store diffs. It just stores the full content of every file every time it’s modified and committed. I work on concurrent editing systems (using CRDTs). People always ask about how we deal with the problem that if we store changes forever, the filesize grows indefinitely. Modern CRDTs are massively more space efficient than git but nobody seems to notice or care about the size of our git repositories. I think git supports shallow clones but I don’t know anyone who bothers with them.

mandeepj 4y ago
> It just stores the full content of every file every time it’s modified and committed. That's not true; otherwise each git branch folder would be huge, but it's not. It stores the snapshot of changes in branch folders
- thro388 4y ago
  Maybe OP stores binary files?
  - sekao 4y ago
    Binary files are also delta'd in git.
- _ikke_ 4y ago
  > It stores the snapshot of changes in branch folders No, gp is right. There is no snapshot of changes. Each unique version of a file is an object in the object store. That does mean each commit would only introduce new objects for files that have changed (these are already added to the object store when you use `git add`) . But the pack files like the article explains (which are created regularly during gc/maintenance runs) is why the repo size can often be smaller then the working tree. > Not only is each object compressed individually, they can also be compressed against each other to take advantage of common data.
- CorrectHorseBat 4y ago
  Branches are just pointers to commits in Git. How those commits are stored doesn't affect their size.
tomstuart 4y ago
I know what you mean, but — as the article explains [0] — Git’s packfile format uses delta compression, which is essentially “storing diffs”. [0] https://github.blog/2022-08-29-gits-database-internals-i-packed-object-store/#delta-compression https://github.blog/2022-08-29-gits-database-internals-i-pac...
- srvmshr 4y ago
  In my mental model of git internals: I envision the initial commit stores a full copy in object store - and thereafter the states are captured as diffs. If a new file is added at a later commit stage, of course the "blob" will be stored, but subsequent commits will add the deltas to it - not a full copy.
  - masklinn 4y ago
    That's not the case. The actual model of git internals is that every commit always stores full copies (of modified entries, unmodified entries are dedup'd through the magic of content addressing). Then as an optimisation of the packing process git will perform delta compression between objects based on a bunch of heuristics. Until you pack you get full copies, and depending how "hard" you pack you may or may not gets huge savings. Furthermore, Git's pack files are ordered, so depending on the selection it makes it can diff later files from earlier ones (your model) or the reverse, or even both at the same time. You can see some of the early packing heuristics at https://github.com/git/git/blob/master/Documentation/technical/pack-heuristics.txt https://github.com/git/git/blob/master/Documentation/technic...
    - cryptonector 4y ago
      > That's not the case. As far as the actual internals are concerned, yes. But as a model it's not wrong. It's like epicycles aren't wrong as a model either. GP's mental model of Git is mine as well, and it works very well for me.
      - masklinn 4y ago
pocketarc 4y ago
It's a reminder that sometimes it's OK to ignore hypothetical problems. "How does git deal with the problem that if we store changes forever, the file size grows indefinitely?" Even though Git uses deltas for diffs to reduce space usage, it still technically grows indefinitely, same as CRDTs. So the answer is it doesn't, and it doesn't matter. If the size of something actually becomes a problem, -then- it can be handled (e.g. Git LFS). With CRDTs, depending on what you're doing, it might never become a problem. Keeping all versions of a document is entirely inconsequential, as an example. The size might grow indefinitely but how many edits is a document going to have, realistically?
MonkeyMalarky 4y ago
I have used shallow clones to speed up and shrink what gets pulled into a docker image when building it. Of course it was a hack to make up for past mistakes but I was glad the option was available.
- masklinn 4y ago
  Did you need to update the docker image in-place afterwards? Unless you did, why not just export?
Existenceblinks 4y ago
It kinda store diff in `rr-cache/` the preimage | postimage | thisimage combined is a kind of diff.
Nzen 4y ago
Ah, thanks. You've reminded me about the 2016 debacle [0] wherein github rate limited the cocoapods repository because their practice of using a shallow clone and later fetches impacted github's performance. I'm going to warc that now, so I don't forget again. [0] https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomment-193772935 https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomm... [+] https://github.com/orgs/Homebrew/discussions/225 https://github.com/orgs/Homebrew/discussions/225
giancarlostoro 4y ago
It's easy to forget that code is... so small, but I guess if your codebase on its own was hundreds of megabytes big, then that would become noticed more quickly. I never realized it stored a copy every time, I assumed it used diffs. So git uses the same solution some of us use when we're about to do something on git we're unfamiliar with... you copy the entire directory housing your code + git repo into a new location just in case it fails miserably.
- NavinF 4y ago
  > I never realized it stored a copy every time, I assumed it used diffs git does store diffs. People are confusing the logical representation with the on-disk layout.
masklinn 4y ago
> Modern CRDTs are massively more space efficient than git but nobody seems to notice or care about the size of our git repositories. The packing process explained in the article is why: rather than make compression an intrinsic part of the system's model (which most VCS do / have done), git moves it to a later "optimisation" phase. This is an additional and somewhat odd costs, but on the other hand it can be extremely efficient (at $DAYJOB we've got files whose compression ratio is 99%). An other interesting bit (which makes it very annoying to inspect the raw contents of a git repo) is that git DEFLATEs object contents when it stores them on-disk. But while that's clearly documented for packfiles (although the lack of end-of-data marker is implicit and extremely annoying), it's not really spelled out for loose object, you get to find out when you can't make heads or tails of loose objects, and git barfs at your hand-crafted loose objects.
HWR_14 4y ago
I still don't grok how CRDTs deal with conflicts. Or is it just "this happens so rarely I'll just use last update"?
- davidjfelix 4y ago
  That's a LWW (last write wins) resolution strategy. Here's a pretty good overview of a variety of CRDT methods: https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type#Known_CRDTs https://en.wikipedia.org/wiki/Conflict-free_replicated_data_...

mandeepj 4y ago

> It just stores the full content of every file every time it’s modified and committed. That's not true; otherwise each git branch folder would be huge, but it's not. It stores the snapshot of changes in branch folders

thro388 4y ago
Maybe OP stores binary files?
- sekao 4y ago
  Binary files are also delta'd in git.
_ikke_ 4y ago
> It stores the snapshot of changes in branch folders No, gp is right. There is no snapshot of changes. Each unique version of a file is an object in the object store. That does mean each commit would only introduce new objects for files that have changed (these are already added to the object store when you use `git add`) . But the pack files like the article explains (which are created regularly during gc/maintenance runs) is why the repo size can often be smaller then the working tree. > Not only is each object compressed individually, they can also be compressed against each other to take advantage of common data.
CorrectHorseBat 4y ago
Branches are just pointers to commits in Git. How those commits are stored doesn't affect their size.

tomstuart 4y ago

I know what you mean, but — as the article explains [0] — Git’s packfile format uses delta compression, which is essentially “storing diffs”. [0] https://github.blog/2022-08-29-gits-database-internals-i-packed-object-store/#delta-compression https://github.blog/2022-08-29-gits-database-internals-i-pac...

srvmshr 4y ago
In my mental model of git internals: I envision the initial commit stores a full copy in object store - and thereafter the states are captured as diffs. If a new file is added at a later commit stage, of course the "blob" will be stored, but subsequent commits will add the deltas to it - not a full copy.
- masklinn 4y ago
  That's not the case. The actual model of git internals is that every commit always stores full copies (of modified entries, unmodified entries are dedup'd through the magic of content addressing). Then as an optimisation of the packing process git will perform delta compression between objects based on a bunch of heuristics. Until you pack you get full copies, and depending how "hard" you pack you may or may not gets huge savings. Furthermore, Git's pack files are ordered, so depending on the selection it makes it can diff later files from earlier ones (your model) or the reverse, or even both at the same time. You can see some of the early packing heuristics at https://github.com/git/git/blob/master/Documentation/technical/pack-heuristics.txt https://github.com/git/git/blob/master/Documentation/technic...
  - cryptonector 4y ago
    > That's not the case. As far as the actual internals are concerned, yes. But as a model it's not wrong. It's like epicycles aren't wrong as a model either. GP's mental model of Git is mine as well, and it works very well for me.
    - masklinn 4y ago

pocketarc 4y ago

It's a reminder that sometimes it's OK to ignore hypothetical problems. "How does git deal with the problem that if we store changes forever, the file size grows indefinitely?" Even though Git uses deltas for diffs to reduce space usage, it still technically grows indefinitely, same as CRDTs. So the answer is it doesn't, and it doesn't matter. If the size of something actually becomes a problem, -then- it can be handled (e.g. Git LFS). With CRDTs, depending on what you're doing, it might never become a problem. Keeping all versions of a document is entirely inconsequential, as an example. The size might grow indefinitely but how many edits is a document going to have, realistically?

MonkeyMalarky 4y ago

I have used shallow clones to speed up and shrink what gets pulled into a docker image when building it. Of course it was a hack to make up for past mistakes but I was glad the option was available.

masklinn 4y ago
Did you need to update the docker image in-place afterwards? Unless you did, why not just export?

Existenceblinks 4y ago

It kinda store diff in `rr-cache/` the preimage | postimage | thisimage combined is a kind of diff.

Nzen 4y ago

Ah, thanks. You've reminded me about the 2016 debacle [0] wherein github rate limited the cocoapods repository because their practice of using a shallow clone and later fetches impacted github's performance. I'm going to warc that now, so I don't forget again. [0] https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomment-193772935 https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomm... [+] https://github.com/orgs/Homebrew/discussions/225 https://github.com/orgs/Homebrew/discussions/225

giancarlostoro 4y ago

It's easy to forget that code is... so small, but I guess if your codebase on its own was hundreds of megabytes big, then that would become noticed more quickly. I never realized it stored a copy every time, I assumed it used diffs. So git uses the same solution some of us use when we're about to do something on git we're unfamiliar with... you copy the entire directory housing your code + git repo into a new location just in case it fails miserably.

NavinF 4y ago
> I never realized it stored a copy every time, I assumed it used diffs git does store diffs. People are confusing the logical representation with the on-disk layout.

masklinn 4y ago

> Modern CRDTs are massively more space efficient than git but nobody seems to notice or care about the size of our git repositories. The packing process explained in the article is why: rather than make compression an intrinsic part of the system's model (which most VCS do / have done), git moves it to a later "optimisation" phase. This is an additional and somewhat odd costs, but on the other hand it can be extremely efficient (at $DAYJOB we've got files whose compression ratio is 99%). An other interesting bit (which makes it very annoying to inspect the raw contents of a git repo) is that git DEFLATEs object contents when it stores them on-disk. But while that's clearly documented for packfiles (although the lack of end-of-data marker is implicit and extremely annoying), it's not really spelled out for loose object, you get to find out when you can't make heads or tails of loose objects, and git barfs at your hand-crafted loose objects.

HWR_14 4y ago

I still don't grok how CRDTs deal with conflicts. Or is it just "this happens so rarely I'll just use last update"?

davidjfelix 4y ago
That's a LWW (last write wins) resolution strategy. Here's a pretty good overview of a variety of CRDT methods: https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type#Known_CRDTs https://en.wikipedia.org/wiki/Conflict-free_replicated_data_...