3 ms·

I explored this idea back in the day with darcs (though I don't think darcs' proposed character-based patch format "chunks" ever made it past the proposal stage

I explored this idea back in the day with darcs (though I don't think darcs' proposed character-based patch format "chunks" ever made it past the proposal stage), as some back and forth discussions led me to what I think is a perfect compromise on power, time (to compute), and universality. I explored using an off the shelf syntax highlighting tokenizer (Pygments specifically as a it was a Python prototype) [1] to build character-based diffs. Got surprisingly good results from what I tested. Most off the shelf syntax highlighters support a huge variety of languages. One advantage to using a syntax highlighting tokenizer/lexer is that they are designed to be fast (many of them need to be real time in text editors after all), and handle "degenerate cases" very well (text editors spend a lot of time on work in progress code that doesn't fully describe a complete compilable unit; version control systems just as well need to be good at storing work in progress states). The finished prototype I built created standard character-based unidiffs, and it sounds like could possibly be slotted directly into Pijul now. I'd be very interested to experiment further with it now that one of the patch algebra VCSes has finally adopted a "chunk" option. [1] https://github.com/WorldMaker/tokdiff https://github.com/WorldMaker/tokdiff (ETA: Suppose the first step is that it is time to get the band back together and update the prototype to Python 3.)

chriswarbo 6y ago
Very cool. I suppose these days tree-sitter would be one way to support a lot of languages at once ( https://tree-sitter.github.io/tree-sitter/#available-parsers https://tree-sitter.github.io/tree-sitter/#available-parsers )
- WorldMaker 6y ago
  Still a tiny fraction compared to available tokenizers in Pygments: https://pygments.org/languages/ https://pygments.org/languages/ (Or the vast number of TextMate grammars in the wild, such as editors like VS Code have standardized on.) The "universality" of tokenizers is tough to beat, and though the tokenizers seem "dumber" than parser brethren, it's at a nice maxima for "good enough" for things like character-based diffs.
pmeunier 6y ago
I love that. > The finished prototype I built created standard character-based unidiffs, and it sounds like could possibly be slotted directly into Pijul now. The diff algorithm, acting on lines, is pretty generic in Pijul, but there is a crucial and nontrivial pass after that, to get from a line diff to a graph diff.

chriswarbo 6y ago

Very cool. I suppose these days tree-sitter would be one way to support a lot of languages at once ( https://tree-sitter.github.io/tree-sitter/#available-parsers https://tree-sitter.github.io/tree-sitter/#available-parsers )

WorldMaker 6y ago
Still a tiny fraction compared to available tokenizers in Pygments: https://pygments.org/languages/ https://pygments.org/languages/ (Or the vast number of TextMate grammars in the wild, such as editors like VS Code have standardized on.) The "universality" of tokenizers is tough to beat, and though the tokenizers seem "dumber" than parser brethren, it's at a nice maxima for "good enough" for things like character-based diffs.

pmeunier 6y ago

I love that. > The finished prototype I built created standard character-based unidiffs, and it sounds like could possibly be slotted directly into Pijul now. The diff algorithm, acting on lines, is pretty generic in Pijul, but there is a crucial and nontrivial pass after that, to get from a line diff to a graph diff.