4 ms·

Why don’t we just check ASTs in to git instead of text files ?

by kristo 6y ago

ssivark 6y ago
I think this is the key question, and doing this will enable numerous interface improvements for people working with software (Eg: separation of content and presents means that everybody can use their favorite view — three-space-tabs, extra-long lines, whatever). The challenge is that version control on text files (list of lines) is fairly straight forward and language agnostic, while OTOH the VCS would have to run after the language parser processes on the text file, and then be able to interface with the AST generated by the parser (for multiple languages). I presume this would become easy if different languages have the same parsing setup, or we have language/project-specific VCS tools. While the answers are not obvious (to me, at least) this definitely seems like the right direction to move in, instead of getting hung up on plain text and formatting issues.
- nyanpasu64 6y ago
  Storing an AST instead of text might be a way to eliminate parsing ambiguity of angle brackets for generic arguments (as opposed to comparisons). It might still be visually ambiguous, but I don't think it's a problem in practice (if the display forces whitespace around comparisons)?
i_dursun 6y ago
I believe unison does something along those lines: https://www.unisonweb.org/ https://www.unisonweb.org/
tlrobinson 6y ago
There are some cases where formatters like Prettier don't have one canonical representation for an AST, in particular behavior around line breaks (multiple empty lines are allowed, a line break after an "{" opening brace in an object literal will force it to span multiple lines, etc) But yeah, otherwise everyone could just have their own preferred formatter/settings and we could commit whatever representation is most optimal for diffing to version control (e.x. multi-line object literals, trailing commas in arrays/args, etc). You could use something more compact like a binary AST but for compatibility with "legacy" tools that deal with text I think it would be better to use a more verbose text-based representation.
pmiller2 6y ago
Because that requires either compiler support, or a plug-in to be written for each language and language version. But, I guess you can always fall back to text files in the worst case.
cjohansson 6y ago
Sounds like a great idea, or binary representation of lexer tokens, the AST is larger in size than the lexer tokens and the AST can be easily generated from the tokens
jyounker 6y ago
I believe that's called "lisp".
- Firadeoclus 6y ago
  Which lisps don't use text files for source code?
philsnow 6y ago
I have thought about this a bit in the past, and I think the answer is that the AST for a given blob of text can be different depending on which version of the language you interpret it with. consider 'print "foo"' in python 2.7: Python 2.7.15 (default, Feb 14 2019, 16:26:56) [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.10.44.4)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import ast >>> ast.dump(ast.parse('print "foo"')) "Module(body=[Print(dest=None, values=[Str(s='foo')], nl=True)])" but in python 3.7: Python 3.7.2 (default, Feb 14 2019, 16:12:42) [Clang 10.0.0 (clang-1000.10.44.4)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import ast >>> ast.parse('print "foo"') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/philsnow/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 35, in parse return compile(source, filename, mode, PyCF_ONLY_AST) File "<unknown>", line 1 print "foo" ^ SyntaxError: Missing parentheses in call to 'print'. Did you mean print("foo")? nothing changed in the text, but the AST is different. What should get committed to this hypothetical "git-ast"? The answer probably lies in creating a different abstraction , different building blocks. In git we have bytes -> blobs -> trees -> commits, where the bytes of a given file get annotated with a file name (and mode bits etc etc, gross posix things) by the tree object. In git-ast, maybe the abstraction is bytes -> lexemes (which annotates the bytes with a language version, and thus lexer version) -> tree? I haven't gone down the thought experiment that far.
- jlarocco 6y ago
  I suspect you're probably right, but that example isn't the greatest because Python 3 isn't source compatible with Python 2. The source code text itself isn't even backwards compatible in that case.
  - philsnow 6y ago
    That was maybe not a great example, pick any language change where a new keyword was introduced that wasn't previously a keyword or reserved word. I tried to make an example with `with`, because that was introduced between 2.4 and 2.5 IIRC, and 2.4 and 2.5 aren't nearly as far apart as 2.* and 3.*, but I didn't find an easy way to install a 2.4 python (it didn't compile in pyenv on my mac, and I looked at a couple docker images before giving up).
mook 6y ago
I kind of like the ability to have comments, though. The tools I've worked with have issues round-tripping them though ASTs.

Why don’t we just check ASTs in to git instead of text files ?

by kristo 6y ago

Why don’t we just check ASTs in to git instead of text files ?

I think this is the key question, and doing this will enable numerous interface improvements for people working with software (Eg: separation of content and presents means that everybody can use their favorite view — three-space-tabs, extra-long lines, whatever). The challenge is that version control on text files (list of lines) is fairly straight forward and language agnostic, while OTOH the VCS would have to run after the language parser processes on the text file, and then be able to interface with the AST generated by the parser (for multiple languages). I presume this would become easy if different languages have the same parsing setup, or we have language/project-specific VCS tools. While the answers are not obvious (to me, at least) this definitely seems like the right direction to move in, instead of getting hung up on plain text and formatting issues.

nyanpasu64 6y ago
Storing an AST instead of text might be a way to eliminate parsing ambiguity of angle brackets for generic arguments (as opposed to comparisons). It might still be visually ambiguous, but I don't think it's a problem in practice (if the display forces whitespace around comparisons)?

i_dursun 6y ago

I believe unison does something along those lines: https://www.unisonweb.org/ https://www.unisonweb.org/

tlrobinson 6y ago

There are some cases where formatters like Prettier don't have one canonical representation for an AST, in particular behavior around line breaks (multiple empty lines are allowed, a line break after an "{" opening brace in an object literal will force it to span multiple lines, etc) But yeah, otherwise everyone could just have their own preferred formatter/settings and we could commit whatever representation is most optimal for diffing to version control (e.x. multi-line object literals, trailing commas in arrays/args, etc). You could use something more compact like a binary AST but for compatibility with "legacy" tools that deal with text I think it would be better to use a more verbose text-based representation.

pmiller2 6y ago

Because that requires either compiler support, or a plug-in to be written for each language and language version. But, I guess you can always fall back to text files in the worst case.

cjohansson 6y ago

Sounds like a great idea, or binary representation of lexer tokens, the AST is larger in size than the lexer tokens and the AST can be easily generated from the tokens

jyounker 6y ago

I believe that's called "lisp".

Firadeoclus 6y ago
Which lisps don't use text files for source code?

philsnow 6y ago

I have thought about this a bit in the past, and I think the answer is that the AST for a given blob of text can be different depending on which version of the language you interpret it with. consider 'print "foo"' in python 2.7: Python 2.7.15 (default, Feb 14 2019, 16:26:56) [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.10.44.4)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import ast >>> ast.dump(ast.parse('print "foo"')) "Module(body=[Print(dest=None, values=[Str(s='foo')], nl=True)])" but in python 3.7: Python 3.7.2 (default, Feb 14 2019, 16:12:42) [Clang 10.0.0 (clang-1000.10.44.4)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import ast >>> ast.parse('print "foo"') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/philsnow/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 35, in parse return compile(source, filename, mode, PyCF_ONLY_AST) File "<unknown>", line 1 print "foo" ^ SyntaxError: Missing parentheses in call to 'print'. Did you mean print("foo")? nothing changed in the text, but the AST is different. What should get committed to this hypothetical "git-ast"? The answer probably lies in creating a different abstraction , different building blocks. In git we have bytes -> blobs -> trees -> commits, where the bytes of a given file get annotated with a file name (and mode bits etc etc, gross posix things) by the tree object. In git-ast, maybe the abstraction is bytes -> lexemes (which annotates the bytes with a language version, and thus lexer version) -> tree? I haven't gone down the thought experiment that far.

jlarocco 6y ago
I suspect you're probably right, but that example isn't the greatest because Python 3 isn't source compatible with Python 2. The source code text itself isn't even backwards compatible in that case.
- philsnow 6y ago
  That was maybe not a great example, pick any language change where a new keyword was introduced that wasn't previously a keyword or reserved word. I tried to make an example with `with`, because that was introduced between 2.4 and 2.5 IIRC, and 2.4 and 2.5 aren't nearly as far apart as 2.* and 3.*, but I didn't find an easy way to install a 2.4 python (it didn't compile in pyenv on my mac, and I looked at a couple docker images before giving up).

mook 6y ago

I kind of like the ability to have comments, though. The tools I've worked with have issues round-tripping them though ASTs.