In my investigations ages ago [1] I felt the trick was to go lower that an AST. ASTs by nature generally have to be language-aware and vary so much from languag

In my investigations ages ago [1] I felt the trick was to go lower that an AST. ASTs by nature generally have to be language-aware and vary so much from language to language that trying to generally diff them is rough. I didn't solve the language-aware part, but I did have some really good luck with using tokenizers intended for syntax highlighting. Because they are intended for syntax highlighting they are fast, efficient, and generally work well with "malformed"/in-progress works (which is what you want for source control where saving in progress steps can be important/useful/necessary). It still needs to be language-aware to know which token grammar to use, but syntax highlighting as a field has a relatively well defined shared vocabulary of output token types, which lends to some flexibility in changing the language on the fly with somewhat minimal shifts (particularly things like JS to TS where the base grammars share a lot of tokens). I didn't do much more with it than generate simple character-based diffs that seemed like improvements of comparative line-based diffs, but I got interesting results in my experiments and beat some simple benchmarks in comparing to other character-based diff tools of the time. (That experiment was done in the context of darcs exploring character-based diffs as a way to improve its CRDT-like source control. I still don't think darcs has the proposed character-based patch type. In theory, I could update the experiment and attempt to use it as a git mergetool, but I don't know if it provides as many benefits as a git mergetool than it might in a patch theory VCS like darcs or pijul.) [1] https://github.com/WorldMaker/tokdiff https://github.com/WorldMaker/tokdiff