4 ms·

So, I didn't believe you because my internal mental model of a git repo is a DAG of changesets. And indeed, that is a good way to think about it, because almost

So, I didn't believe you because my internal mental model of a git repo is a DAG of changesets. And indeed, that is a good way to think about it, because almost all of git's operations behave like this. Commit history is almost always presented to the user as a series of diffs. But, you are correct. Internally, git stores the full contents of files and computes the diffs on the fly. For others' benefit, if you want to test for yourself, create a new repo, add a file and make a series of commits with changes. Git objects are compressed with DEFLATE (zlib) so gunzip and unzip won't work. I used https://github.com/jezell/zlibber https://github.com/jezell/zlibber because I was too lazy to write my own quick zlib wrapper. Then doing for o in .git/objects/*/*; do cat "$o" | inflate ; echo ""; done lists the contents of all the git objects. Notice that there are no changesets, only full copies of the file you modified at different states. This was surprising to me, since I had a very different model mentally. I still think the DAG of diffs is the better model mentally, but it is worth understanding that this is not what git is actually doing under the hood. It explains issues that arise doing rebases, cherry-picks, etc. I now also understand the motivation behind Pijul. If I understand correctly, Pijul does use a collection of changesets as the underlying model. Like you say, that can be a critical difference.

kzrdude 10y ago
That is the most basic storage format in git, but it has packfiles too where it uses deltas. But not necessarily changesets' diffs!
dahart 10y ago
I thought git stored diffs too. But I don't understand what difference it makes, isn't the commit storage format an implementation detail? If you want to get to B from A, then I can store B or store B-A. When I have B and want to show a diff, I can calculate B-A. When I have B-A and want to show a diff, it's a no-op. When I have B and want to work, it's a no-op, but when I have B-A and want to work, I have to populate the working tree with A + B-A. It doesn't feel like there's a fundamental difference in the model either way. But if you don't have A in the first place, then there's a real problem, how to take B-A and reconstruct B, without knowing what A is. You have to find A, and there may be multiple acceptable A's. It seems like the fundamental difference here would be not having a strict "parent" for any given patch. I can see why that would make some workflows a little nicer, not being forced to rebase, but I don't see any massive advantages -- as a user what does this really buy me? Does it enable some things like that are impossible with git? Or does it mainly make some advanced git workflows easier?
- lvh 10y ago
  The difference is more than just the storage format. You could compute snapshots on demand and just store the diffs (that'd be impossibly slow, but you could), but you still would have git's DAG instead of darcs' theory of patches, with all of the consequences of that. (I'd explain them here, but I've already explained it in sibling comments on this thread).
- jdbernard 10y ago
  Maybe lvh or one of the others who has more experience with darcs of Pijul will chime in. They've probably spent more time thinking about this. One difference is that by storing the patches only you can understand more clearly what the intended change was. When you store the whole file it is easy to compute the difference between A and B, but may be impossible to compute the correct differences between A, B, and C. By storing the whole file you now have to consider all the possible differences between them, not just the ones introduced by the commits you are trying to merge. I would have to play around with it, but I know there are scenarios involving rebase, revert, and cherry-picking commits that can cause trouble in git that I now understand comes because of the fact that git is storing contents, not diffs. One that I've run into regularly is cherry-picking commits from a dev branch into a master branch to hot-fix bug fixes directly into a prod release instead of waiting until dev gets merged as part of our regular process. If I had commit A on dev and cherry-pick it to master it creates a totally new commit A-1 that becomes part of the history of master. We lost the fact that A and A-1 represent the exact same changeset. Depending on what the changes are, and what further changes happen on dev afterwards, this can cause failed merges requiring manual resolution when dev does finally get merged into master. I imagine that would not be a problem for Pijul.
  - lvh 10y ago
    I think you've hit the nail on the head :)
    - dahart 10y ago
      Those reasons make sense. Would it be fair to summarize all that as "better merging with fewer merge conflicts"? Git certainly has some room to improve in the merge conflict department. I looked at the bad merge example you posted -- I suspect I've hit that before. It's rare, but yeah it's there. I also frequently notice that git complains about merge conflicts, while the custom diff tool I use to resolve them says there's no conflict, and I don't actually have to do anything. Good reason to use a custom merge tool with git. But, given all this, is this really all an outcome of patches vs snapshots, or is this just git's merge algorithm being suboptimal? Certainly git could selectively ignore the DAG when merging, couldn't it? Even after reading the other comments here, it still seems to me like git has more information when merging than the "patch-based" workflow of darcs & Pijul. It seems to me like there's a language problem with trying to draw a distinction between patches and snapshots. Git is still storing and transferring patches at the tree level, even if it's not happening at the file level. Git does not store a commit as a zip snapshot of the entire tree, the commit is still only the changed files. It would be fair (but not standard or common) to call the overlay of changed files a "patch" or a "diff". People do still use git format-patch, and email git "patches" to each other. So it's inherently confusing & problematic to talk about git and say that it doesn't use patches. What does make sense to me is the distinction of having a strict DAG vs not having one -- is that actually what people mean when they talk about snapshots vs patches? Am I tripping on it because I'm being too pedantic about what a "patch" is?
lvh 10y ago
What other operations behave as if it's at its core diffs and not snapshots? merging, rebasing, committing... all operate on refs. You might think you're transplanting changes (and you are), but the inputs are refs, and the outputs are refs. As you mentioned, refs are unambiguously snapshots.

kzrdude 10y ago

That is the most basic storage format in git, but it has packfiles too where it uses deltas. But not necessarily changesets' diffs!

dahart 10y ago

I thought git stored diffs too. But I don't understand what difference it makes, isn't the commit storage format an implementation detail? If you want to get to B from A, then I can store B or store B-A. When I have B and want to show a diff, I can calculate B-A. When I have B-A and want to show a diff, it's a no-op. When I have B and want to work, it's a no-op, but when I have B-A and want to work, I have to populate the working tree with A + B-A. It doesn't feel like there's a fundamental difference in the model either way. But if you don't have A in the first place, then there's a real problem, how to take B-A and reconstruct B, without knowing what A is. You have to find A, and there may be multiple acceptable A's. It seems like the fundamental difference here would be not having a strict "parent" for any given patch. I can see why that would make some workflows a little nicer, not being forced to rebase, but I don't see any massive advantages -- as a user what does this really buy me? Does it enable some things like that are impossible with git? Or does it mainly make some advanced git workflows easier?

lvh 10y ago
The difference is more than just the storage format. You could compute snapshots on demand and just store the diffs (that'd be impossibly slow, but you could), but you still would have git's DAG instead of darcs' theory of patches, with all of the consequences of that. (I'd explain them here, but I've already explained it in sibling comments on this thread).
jdbernard 10y ago
Maybe lvh or one of the others who has more experience with darcs of Pijul will chime in. They've probably spent more time thinking about this. One difference is that by storing the patches only you can understand more clearly what the intended change was. When you store the whole file it is easy to compute the difference between A and B, but may be impossible to compute the correct differences between A, B, and C. By storing the whole file you now have to consider all the possible differences between them, not just the ones introduced by the commits you are trying to merge. I would have to play around with it, but I know there are scenarios involving rebase, revert, and cherry-picking commits that can cause trouble in git that I now understand comes because of the fact that git is storing contents, not diffs. One that I've run into regularly is cherry-picking commits from a dev branch into a master branch to hot-fix bug fixes directly into a prod release instead of waiting until dev gets merged as part of our regular process. If I had commit A on dev and cherry-pick it to master it creates a totally new commit A-1 that becomes part of the history of master. We lost the fact that A and A-1 represent the exact same changeset. Depending on what the changes are, and what further changes happen on dev afterwards, this can cause failed merges requiring manual resolution when dev does finally get merged into master. I imagine that would not be a problem for Pijul.
- lvh 10y ago
  I think you've hit the nail on the head :)
  - dahart 10y ago
    Those reasons make sense. Would it be fair to summarize all that as "better merging with fewer merge conflicts"? Git certainly has some room to improve in the merge conflict department. I looked at the bad merge example you posted -- I suspect I've hit that before. It's rare, but yeah it's there. I also frequently notice that git complains about merge conflicts, while the custom diff tool I use to resolve them says there's no conflict, and I don't actually have to do anything. Good reason to use a custom merge tool with git. But, given all this, is this really all an outcome of patches vs snapshots, or is this just git's merge algorithm being suboptimal? Certainly git could selectively ignore the DAG when merging, couldn't it? Even after reading the other comments here, it still seems to me like git has more information when merging than the "patch-based" workflow of darcs & Pijul. It seems to me like there's a language problem with trying to draw a distinction between patches and snapshots. Git is still storing and transferring patches at the tree level, even if it's not happening at the file level. Git does not store a commit as a zip snapshot of the entire tree, the commit is still only the changed files. It would be fair (but not standard or common) to call the overlay of changed files a "patch" or a "diff". People do still use git format-patch, and email git "patches" to each other. So it's inherently confusing & problematic to talk about git and say that it doesn't use patches. What does make sense to me is the distinction of having a strict DAG vs not having one -- is that actually what people mean when they talk about snapshots vs patches? Am I tripping on it because I'm being too pedantic about what a "patch" is?

lvh 10y ago

What other operations behave as if it's at its core diffs and not snapshots? merging, rebasing, committing... all operate on refs. You might think you're transplanting changes (and you are), but the inputs are refs, and the outputs are refs. As you mentioned, refs are unambiguously snapshots.