7 ms·
One issue with the "diff" model is that in GIT (and most all VCs these days) there is no "one true answer" to the question "how did this old file change to beco
by DannyBee 3y ago
One issue with the "diff" model is that in GIT (and most all VCs these days) there is no "one true answer" to the question "how did this old file change to become this new one".
Instead, what is stored is "what is the smallest/fastest/simplest way to create new file from old file".
(depends on tradeoff)
Which is not the same as what you may have actually done to change old file into new one.
One result is that the textual representation you see in "diff" is just one interpretation of how the file changed - since it is reconstructing it from first principles.
I mention this because the author mention the following positive of the diff mental model:
"most of the time I’m concerned with the change I’m making – if I’m just changing 1 line of code, obviously I’m mostly thinking about just that 1 line of code and not the entire current state of the codebase
when you click on a Git commit on GitHub or use git show, you see the diff, so it’s just what I’m used to seeing"
Of course, you see "a diff", not "the diff", because there is no "the diff". and when you change 1 line of code, the diff may or may not show it as having changed one line of code.
It is likely to in most cases because that usually meets the definition of longest common subsequence that common diff algorithms optimize for.
But not always.
Which leads to interest results when you were thinking about a change one way, but the diff doesn't show what you actually did to the code in your editor :)
- samus 3y agoGit stores states of the repo across time. The diff is something created for the user. Internally, Git might or might store it in an optimized way, but not necessarily.
- dclowd9901 3y agoOk so I think I at least understand now that “git diff” isn’t just showing me some stored information in the repo’s metadata. But still, I can’t reason about how so many changes to a repo can be recorded in a way that’s so efficient that it’s almost imperceptible in time cost. And I’ve yet to see a satisfactory answer to this question. I need concrete examples.
- federiconafria 3y agoGit does not store changes it stores a tree with the whole state of the repo for every commit. This tree has references to the files, when a file is changed a new copy is always added and the new commit will reference the new tree with the reference to the new file. When you ask git for a diff it compares the trees and when there is a difference it compares the files the trees reference and it shows you a diff. Git does not store diffs, it stores file trees.
- samus 3y agoGit is basically an object storage. Most objects in Git are blobs identified by their hash code. For example, a commit points to a directory. A directory points to all files and directories it contains. With this foundation, Git is already quite efficient in common scenarios since most commits create only a few new objects: the commit, the new versions of the files, and all directory nodes all the way to the top. Git could store all of these in the filesystem. Often, they are stored in indexed and compressed pack files though. But the underlying principle remains the same.
- cerved 3y agoIf you do git add . && git commit -m wip then git first checks which files have changed, taking the shortcut of checking if the files mtime is newer than the current index. This is basically the speed of find . For files that are changed it creates new trees and store it in the object store, this is cp + SHA-1 Then it takes all that metadata, comine it to one, add it to a commit message and it's done. Doesn't take a lot of time to do these things, but it will for very large files. Going the other direction is fast, reverse lookup all that metadata and copy the contents back into the working directory
- PhilipRoman 3y agoThis random stackexchange answer is one of my favorite explanations of Git storage model: https://cs.stackexchange.com/a/149300/153403 https://cs.stackexchange.com/a/149300/153403
- nextaccountic 3y ago> But still, I can’t reason about how so many changes to a repo can be recorded in a way that’s so efficient that it’s almost imperceptible in time cost. It's because git does heavy deduplication, in the same way that functional, persistent data structures do heavy deduplication. For example, if you have a tree data structure and one tree (let's call it tree 1) like this A / \ C B And a tree 2 like this A / \ B D Where A is a root node, and B, C and D are subtrees (which may be huge; maybe many megabytes each), what actually gets stored for tree 1 is something like this: A_tree_1 / \ C B And for tree 2: A_tree_2 / \ B D And ultimately when you store both, you have this: A_tree_1 A_tree_2 / \ / \ / \ / \ / \ / \ C B D Ok that's not the best drawing but I think I got the point across: we managed to have two different trees (tree 1 and tree 2) that each share an identical subtree (the B subtree), and as such B needs to be stored only once. (But note that the "spine" of the tree - the path from the possibly deduplicated subtrees until the root - will not itself be deduplicated. That is, even if A is identical between tree 1 and tree 2, you still need different A_tree_1 and A_tree_2, because one of them contains pointers to C and B, and the other contains pointers to B and D) That's how persistent data structures work in functional programming (for example, in Haskell, if you are careful with sharing you can have two different trees share a subtree rather than making a deep copy them, lowering the memory usage) And that's how objects in git work (which include commits, filesystem trees, and file contents, that are called blobs). Two different objects that share subtrees will also share storage. That's not a diff, that's just how things are stored in Git. But ultimately, whenever you need to diff, you only need to consider the diff between C and D; B is common between A_tree_1 and A_tree_2 and doesn't need to be diffed. And the reason this works is that in Git, objects are immutable: you don't modify an existing commit when you do git commit --amend for example, you create an entirely new commit (and the old commit is still in the repository, until you run git gc to get rid of it - but it probably occupies almost no space, because its filesystem tree is probably quite similar to the other commit) (That's how it works in functional programming too: persistent data structures are commonly used there because in functional programming, you don't modify existing trees, you create new ones whenever you want to do a "functional update"; but you deduplicate the trees so that this update can still be somewhat efficient) And the mechanism this deduplication is implemented in Git is content addressing: every object is identified by a sha1 hash of its contents, and every time you try to store an object with a given sha1 hash, git will first verify if this object already exist and if so, it will reuse the existing object. (This means that a "pointer" to a subtree, in git, is just its sha1 hash. So the reason that A_tree_1 and A_tree_2, in Git, need to be different objects, is that inside them there are different hashes: one has C and B, and the other, B and D) (Deduplication in Haskell's data structures wouldn't use content addressing, but instead you manually tell that a given tree will be built with a subtree from another tree, and then it just works because no tree can be modified) Note: content addressing is also how deduplication in filesystems like btrfs work. In those filesystems, the kernel will hash a piece of a file (an extent) and check in a hash table whether that particular extent is already stored somewhere. If yes, you don't need to store again. (you can disable online deduplication too, and do it offline, with a batch job)
- DannyBee 3y agoAgreed, that's my point why the diff model isn't helpful at all to think about.
- ghusbands 3y agoYou claimed the diff was stored - that's what the correction is about. Git's primary storage model is of complete snapshots.
- sho_hn 3y agohttps://tom.preston-werner.com/2009/05/19/the-git-parable.html https://tom.preston-werner.com/2009/05/19/the-git-parable.ht... Tom's 15 years old text remains the only correct, useful, easy to understand explanation of the why and how of git I've ever popularly encountered. Post like the OP's do far more harm than good by introducing mental models comprised of best-effort confused half-knowledge. If (generic) you ever wanted to "really learn git", do yourself a favor and read it.
- MrDarcy 3y agoWhen we ask git to show us a commit, it shows us a diff, because that’s the most helpful model.
- yencabulator 3y agoIf we want to ask git to show us a commit, we need to type git cat-file commit HEAD `git show` does not "show a commit" as such, its man page is pretty clear about that: For commits it shows the log message and textual diff.
- MrDarcy 3y agoThat’s exactly the point. git show HEAD shows the log and textual diff because that’s what matters most from the users perspective. The internal representation doesn’t matter for most users and use cases.
- 3y ago
- gnulinux 3y agoThis is an excellent comment and one should also note that the claim is wrong. If one line is changed "a diff" will likely have two lines: one removed, one added. If you change a sentence "I am studying tonight" to "I am not studying tonight" git diff will show one line deleted when I actually didn't delete anything, conceptually.
- eadler 3y ago`git diff --word-diff` is wonderful for this diff --git 1/old 2/new index 7164d6e..61a276c 100644 --- 1/old +++ 2/new @@ -1 +1 @@ I am {+not+} studying tonight
- yardstick 3y agoWow, why isn’t this the default?!
- _ikke_ 3y agogit diff by default outputs valid patch files that you can feed to patch out git apply. --word-diff does not Kernel development involves working with patches a lot, so it made sense to default to that, and --word-diff was added later anyway.
- yardstick 3y agoAh, makes sense, thanks for the explanation. I’ll have to create an alias for my use!
- dietrichepp 3y ago> This is an excellent comment and one should also note that the claim is wrong. I’ll pretend that this isn’t pedantry. I can understand why you’d say this, but I think you just haven’t explored the way that git diff works, including the configuration options it takes. $ git diff 'HEAD^' diff --git a/file b/file index efb8e76..ca03496 100644 --- a/file +++ b/file @@ -1,3 +1,3 @@ This is context. -I am studying tonight. +I am not studying tonight. This is also context. Try it with --word-diff… $ git diff 'HEAD^' --word-diff diff --git a/file b/file index efb8e76..ca03496 100644 --- a/file +++ b/file @@ -1,3 +1,3 @@ This is context. I am {+not+} studying tonight. This is also context.
- searealist 3y ago> Instead, what is stored is "what is the smallest/fastest/simplest way to create new file from old file". I think this may be misleading to people reading your comment. Git is a snapshot based version control system (Maybe you know this). Each commit contains a complete exact snapshot of the entire repository. On the other hand it does contain optimizations to reduce the size of these snapshots in the form of git packs, and in that sense your comment is technically true, but this is really just an implementation detail.
- krupan 3y agoEven the fact that it's snapshots (but possibly compressed) is an implementation detail. I actually really like the generalized "smallest/fastest/simplest way to create new file from old file" explanation. "Fastest" was a primary goal of git. Same with Mercurial. Other distributed revision control tools that were designed around the same time (bzr, darcs) didn't optimize for speed and you could tell.
- searealist 3y agoI disagree, snapshots is the fundamental model. Otherwise, shallow clones would not be possible.
- usr1106 3y agoOf course they would be possible, just harder and less efficient to implement. The snapshot representation can be transformed into a chain of diffs representation and the other way round without losing anything except performance.
- codeflo 3y ago> I actually really like the generalized "smallest/fastest/simplest way to create new file from old file" explanation. You might like that explanation because it fits a mental model you may have acquired previously, but at the user interface level, Git stores snapshots. It's not an implementation detail, it's what the entire command line is based on.
- deleted 3y ago[deleted]
- hunter2_ 3y ago> when you change 1 line of code, the diff may or may not show it as having changed one line of code. Can you offer an example where an editor is used to change one line of code, and a resulting diff leads the user to believe that the number of changed lines is not one? I don't think people interpret an adjacent opposing pair (one line removed, immediately followed by one line added) as not show[ing] it as having changed one line of code.
- jstanley 3y agoA diff like this is perfectly valid: -foo -bar -baz +foo +bar Although I don't know any tool which would actually generate such a diff.
- hunter2_ 3y agoOh! I was keeping my thoughts scoped to the output of existing tools. Beyond that doesn't seem worth considering in this particular context.
- seba_dos1 3y agoThe existing tools already let you choose the algorithm used to generate the diff, so it's not a big stretch to imagine an algorithm that may do just this. It doesn't make much sense for such 1 line example (although it could if you had, say, an algorithm that considers whole paragraphs, or blocks in some programming language, rather than lines - could be useful for line art perhaps), but with more complex changes there won't be a single "optimal" representation, especially when what's "optimal" depends on what you want to optimize for.
- msla 3y ago> GIT Just out of curiosity: What do you think GIT stands for?
- Biganon 3y agoGust in Time
- bigfishrunning 3y agoGee, It's Torvalds
- formerly_proven 3y agoAs an open source project it needs to be recursive: GIT: It's Torvald's
- keybored 3y agoGIT: GIT Is Torvald’s
- DannyBee 3y agoI don't spend a lot of time editing my comments for word correctness as it's HN, not something that matters. So you will find plenty of occasional misspellings, etc.
- kelnos 3y agoDoes that really matter all that much, though? Often enough, I make a bunch of changes to a repo, but want to split those changes across multiple commits. When I'm staging changes before I make a commit, I'm doing so by looking at the diff (via 'git add -p'). By that point I don't really have any memory of how I made those changes; I don't recall details like "I deleted this line, and then typed this other line in its place", or "I deleted these characters and replaced them with these others", or "I added these characters to an existing line". Ultimately, when I'm looking at it, the diff that git generates is the change, to me. And yes, I get that there's no "the" diff; git could change its algorithm between versions, and the diffs between commits might change (even though the end results of those diffs would be the same). But ultimately I don't really care about that. My overall thoughts about the change are qualitative, and are represented by what I put in the commit message. Whether the diff is represented as "3 deleted lines, then 4 added lines, and finally 1 deleted line", or "2 deleted lines, then 4 added lines, and finally 2 deleted lines" (or whatever) is pretty irrelevant to me.
- 8organicbits 3y agoI think tracking the "how" of a change can help with review. Currently I see 40 lines deleted in this file and 40 added in another. If this was a "cut and paste" then I may not need to review the code as deeply as it's not new, but I'd focus on if it works in the new location.
- hibbelig 3y ago> Which is not the same as what you may have actually done to change old file into new one. I’ve seen this with methods: I add a method, so I expect to see the whole new method in the diff. But actually the diff looks as if I’ve inserted something just before the closing brace of the existing method above the new one. I’ve learned to detect this when reading diffs but a nice display would be, err, nice.
- Sesse__ 3y agoThis is sometimes called a “slider”. Difftastic has special logic to try to deal with sliders and get closer to what you want.
- hibbelig 3y agoOh yes, I need to remember to show diffs using difftastic.
- OJFord 3y agoGit has a few (configurable) diff algorithms; 'histogram' is more readable (IMO, but that's the design goal - or rather is of 'patience' which it extends/speeds up) than the default Myers algorithm, but it does still suffer from the issue you describe IME. Paper comparing Myers and histogram: https://link.springer.com/article/10.1007/s10664-019-09772-z https://link.springer.com/article/10.1007/s10664-019-09772-z Naïvely it seems like it should be not that hard to fix (be less greedy?) or even that it's an example of a case where the algorithm should be less smart: it's such a common case, hardcode paren chars for some kind of exception or second pass. But that is a naïve take, I've never looked at the algorithm or source code at all.
- emmelaich 3y agoI think you want the 'patience' option... --patience Generate a diff using the "patience diff" algorithm.
- yencabulator 3y agoYou'll probably enjoy the patience or histogram diff algorithms. See https://luppeng.wordpress.com/2020/10/10/when-to-use-each-of-the-git-diff-algorithms/ https://luppeng.wordpress.com/2020/10/10/when-to-use-each-of... for more. [diff] algorithm = "histogram" You can also teach git to always show e.g. function names in the diff hunk heading. For example, to always show previous Markdown heading: ~/.config/git/config [diff "markdown"] xfuncname = "^(#+\\s*.*)" ~/.config/git/attributes *.md diff=markdown whitespace=-blank-at-eol
- cxr 3y ago> Of course, you see "a diff", not "the diff", because there is no "the diff" 10–15 years ago when I cared about this sort of thing and I was unhappy with the way that Mercurial diffed two sets of files, I'd take steps just before submitting a patch to edit it by hand so both I and the reviewer had a clearer picture that matched the actual change I was making, in contrast to some goofy-looking and confusing insertion/deletion combo keyed off an arbitrary closing brace or whatever.
- douglee650 3y agoIf you care enough about it, you add a comment and it shows up in the diff?