5 ms·

I would maybe be interested in Git allowing you to plug in your own diff generators for different file types. But I would not want Git itself trying to underst

I would maybe be interested in Git allowing you to plug in your own diff generators for different file types. But I would not want Git itself trying to understand the contents of files. That seems to me to be an idea that lives on a misconception of the "things programmers believe about names" variety. Not every file in source control is source code. Not every programming language's grammar maps to an abstract syntax tree. In some files, such as makefiles, the difference between tabs and spaces is semantically significant. Some languages (such as Fortran and Racket) have variable syntax. And so on and so forth. So I think that we really don't want the source control system itself trying to get too smart about the contents of files. That will inevitably make the source control system less compatible with the various kinds of things you might want to put into source control. And it will also make the source control system a lot more complicated than it would otherwise be, in return for a largely theoretical payoff. But if we want to delegate the work of generating diffs off to other people, so that Git can allow for syntax or semantics-aware diffing without having to personally wade into that quagmire (and perhaps also allowing language communities to support multiple source control systems, a bit like how it works with LSP), that might be an interesting thing to experiment with.

saurik 5y ago
> I would maybe be interested in Git allowing you to plug in your own diff generators for different file types. This is already supported.
- lux 5y ago
  A common example is UnityYAMLMerge for merging the Unity game engine's generated files. https://docs.unity3d.com/Manual/SmartMerge.html https://docs.unity3d.com/Manual/SmartMerge.html Configuring it to work with Git and others is a little ways down the page, but would apply the same for other diff tools.
- franga2000 5y ago
  I looked this up and for anyone wondering, it's called "diff/merge drivers", but there are only a handful of them out there. Some highlights from a few minutes of searching: - MS Office: https://github.com/lcnittl/DMFO https://github.com/lcnittl/DMFO - SQLite: https://github.com/cannadayr/git-sqlite https://github.com/cannadayr/git-sqlite - Jupyter notebooks: https://nbdime.readthedocs.io/en/latest/vcs.html#git-integration https://nbdime.readthedocs.io/en/latest/vcs.html#git-integra... One big caveat of this is that since git doesn't really store just a stack of diffs, despite the fact it presents itself as such to the user, a custom merge driver will not make your .git grow any less than it would normally.
  - est31 5y ago
    > One big caveat of this is that since git doesn't really store just a stack of diffs, despite the fact it presents itself as such to the user, a custom merge driver will not make your .git grow any less than it would normally. Note that git does support using deltas for storage. But according to docs, custom diff drivers aren't used for those, instead it's a instruction based format. https://git-scm.com/docs/pack-format#_deltified_representation https://git-scm.com/docs/pack-format#_deltified_representati...
  - dexen 5y ago
    >One big caveat of this is that since git doesn't really store just a stack of diffs, despite the fact it presents itself as such to the user, >a custom merge driver will not make your .git grow any less than it would normally. You can go a long way with simple homespun methods. SQL dumps delta-compress in Git very well - especially if you tweak the dump options a bit so that the row order is mostly stable. The same most probably goes for any other textual dump - if the (unchanged) content is largely ordered in stable fashion, it will delta-compress very well, even without any explicit, specialized support in Git itself.
- deleted 5y ago
  [deleted]
- shoo 5y ago
  I investigated the option of using a custom git merge driver for a project where we were planning to version control a bunch of data files using git. Here's a proof of concept python merge driver I bashed together at the time to auto-merge JSON objects: https://github.com/fcostin/jsonmerge_git_merge_driver https://github.com/fcostin/jsonmerge_git_merge_driver This never went anywhere near production, but it was very easy to put together something basic. One complication with using a custom merge driver, as discussed by https://github.com/Praqma/git-merge-driver https://github.com/Praqma/git-merge-driver , is that they need to be configured inside the `.git/config` of the repo, which itself is not version controlled. So there's an additional config management overhead to rolling that out to everyone in a machine. Additionally, if outsourcing hosting for git repos, it may not be supported to install and configure a custom merge driver for merges conducted by the hosting platform (e.g. merges created by github.com pull request workflow). One idea I had at the time was using external schema files (e.g. JSON schema for JSON files) to help guide/constrain the result of the merge. I never implemented it, but it should be possible. If the schemas were also version controlled in the same git repo that stores the data, you'd need to figure out which one (and which version) to load when resolving a merge conflict of a data file. There doesn't seem to be a well-supported robust way for a merge driver script to discover the source and destination branches, but there are some potentially fragile ways of doing it that work some of the time.
- pwdisswordfish8 5y ago
  Next you will be telling me that git is already a blockchain!
madmax96 5y ago
I disagree. Many engineers want to refactor across a sequence of small PRs, for example. Small PRs are a good thing, because they’re easier to understand. But today, Git makes this painful. Also, understanding how the meaning of code changes over time can help reduce bugs. The solution will have to be pluggable. But I think it is possible, and there are sane things to do (e.g. fall back to vanilla git) when there are missing plugs.
ffwacom 5y ago
> Not every programming language's grammar maps to an abstract syntax tree Are there some examples of this?
- simcop2387 5y ago
  Perl, this is because you can't actually properly parse Perl without also running Perl code at the same time.
  - justinator 5y ago
    https://metacpan.org/pod/PPI https://metacpan.org/pod/PPI
    - simcop2387 5y ago
      PPI and PPR will parse most perl programs, but the moment you start doing stuff with prototypes or source filters they fail completely.
  - rowanG077 5y ago
    That you can't properly parse Perl without also running Perl doesn't mean it doesn't have an AST. I don't know about Perl so it could very well be that it does not have an AST but your argument doesn't show it does not.
    - simcop2387 5y ago
      Perl code changes how the parsing will be done, both by adding keywords, changing how many arguments the parser will look for, and even arbitrarily changing the language entirely with a source filter. It's largely that last bit that makes creating a consistent or coherent AST so impractical it may as well be impossible.
  - rurban 5y ago
    The AST contains code, but it's still an AST. Every AST contains code. Perl has a dynamic lexer, but still uses a static YACC grammar.
ironmagma 5y ago
Not only that, but imagine you realize there is a bug in the parsing tool. Now you have to go back and re-parse the code, or otherwise just deal with a bad history forever. Suddenly you’re storing text again.

saurik 5y ago

> I would maybe be interested in Git allowing you to plug in your own diff generators for different file types. This is already supported.

lux 5y ago
A common example is UnityYAMLMerge for merging the Unity game engine's generated files. https://docs.unity3d.com/Manual/SmartMerge.html https://docs.unity3d.com/Manual/SmartMerge.html Configuring it to work with Git and others is a little ways down the page, but would apply the same for other diff tools.
franga2000 5y ago
I looked this up and for anyone wondering, it's called "diff/merge drivers", but there are only a handful of them out there. Some highlights from a few minutes of searching: - MS Office: https://github.com/lcnittl/DMFO https://github.com/lcnittl/DMFO - SQLite: https://github.com/cannadayr/git-sqlite https://github.com/cannadayr/git-sqlite - Jupyter notebooks: https://nbdime.readthedocs.io/en/latest/vcs.html#git-integration https://nbdime.readthedocs.io/en/latest/vcs.html#git-integra... One big caveat of this is that since git doesn't really store just a stack of diffs, despite the fact it presents itself as such to the user, a custom merge driver will not make your .git grow any less than it would normally.
- est31 5y ago
  > One big caveat of this is that since git doesn't really store just a stack of diffs, despite the fact it presents itself as such to the user, a custom merge driver will not make your .git grow any less than it would normally. Note that git does support using deltas for storage. But according to docs, custom diff drivers aren't used for those, instead it's a instruction based format. https://git-scm.com/docs/pack-format#_deltified_representation https://git-scm.com/docs/pack-format#_deltified_representati...
- dexen 5y ago
  >One big caveat of this is that since git doesn't really store just a stack of diffs, despite the fact it presents itself as such to the user, >a custom merge driver will not make your .git grow any less than it would normally. You can go a long way with simple homespun methods. SQL dumps delta-compress in Git very well - especially if you tweak the dump options a bit so that the row order is mostly stable. The same most probably goes for any other textual dump - if the (unchanged) content is largely ordered in stable fashion, it will delta-compress very well, even without any explicit, specialized support in Git itself.
deleted 5y ago
[deleted]
shoo 5y ago
I investigated the option of using a custom git merge driver for a project where we were planning to version control a bunch of data files using git. Here's a proof of concept python merge driver I bashed together at the time to auto-merge JSON objects: https://github.com/fcostin/jsonmerge_git_merge_driver https://github.com/fcostin/jsonmerge_git_merge_driver This never went anywhere near production, but it was very easy to put together something basic. One complication with using a custom merge driver, as discussed by https://github.com/Praqma/git-merge-driver https://github.com/Praqma/git-merge-driver , is that they need to be configured inside the `.git/config` of the repo, which itself is not version controlled. So there's an additional config management overhead to rolling that out to everyone in a machine. Additionally, if outsourcing hosting for git repos, it may not be supported to install and configure a custom merge driver for merges conducted by the hosting platform (e.g. merges created by github.com pull request workflow). One idea I had at the time was using external schema files (e.g. JSON schema for JSON files) to help guide/constrain the result of the merge. I never implemented it, but it should be possible. If the schemas were also version controlled in the same git repo that stores the data, you'd need to figure out which one (and which version) to load when resolving a merge conflict of a data file. There doesn't seem to be a well-supported robust way for a merge driver script to discover the source and destination branches, but there are some potentially fragile ways of doing it that work some of the time.
pwdisswordfish8 5y ago
Next you will be telling me that git is already a blockchain!

madmax96 5y ago

I disagree. Many engineers want to refactor across a sequence of small PRs, for example. Small PRs are a good thing, because they’re easier to understand. But today, Git makes this painful. Also, understanding how the meaning of code changes over time can help reduce bugs. The solution will have to be pluggable. But I think it is possible, and there are sane things to do (e.g. fall back to vanilla git) when there are missing plugs.

ffwacom 5y ago

> Not every programming language's grammar maps to an abstract syntax tree Are there some examples of this?

simcop2387 5y ago
Perl, this is because you can't actually properly parse Perl without also running Perl code at the same time.
- justinator 5y ago
  https://metacpan.org/pod/PPI https://metacpan.org/pod/PPI
  - simcop2387 5y ago
    PPI and PPR will parse most perl programs, but the moment you start doing stuff with prototypes or source filters they fail completely.
- rowanG077 5y ago
  That you can't properly parse Perl without also running Perl doesn't mean it doesn't have an AST. I don't know about Perl so it could very well be that it does not have an AST but your argument doesn't show it does not.
  - simcop2387 5y ago
    Perl code changes how the parsing will be done, both by adding keywords, changing how many arguments the parser will look for, and even arbitrarily changing the language entirely with a source filter. It's largely that last bit that makes creating a consistent or coherent AST so impractical it may as well be impossible.
- rurban 5y ago
  The AST contains code, but it's still an AST. Every AST contains code. Perl has a dynamic lexer, but still uses a static YACC grammar.

ironmagma 5y ago

Not only that, but imagine you realize there is a bug in the parsing tool. Now you have to go back and re-parse the code, or otherwise just deal with a bad history forever. Suddenly you’re storing text again.