3 ms·
I have thought about this a bit in the past, and I think the answer is that the AST for a given blob of text can be different depending on which version of the
by philsnow 6y ago
I have thought about this a bit in the past, and I think the answer is that the AST for a given blob of text can be different depending on which version of the language you interpret it with. consider 'print "foo"' in python 2.7:
Python 2.7.15 (default, Feb 14 2019, 16:26:56)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.10.44.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> ast.dump(ast.parse('print "foo"'))
"Module(body=[Print(dest=None, values=[Str(s='foo')], nl=True)])"
but in python 3.7:
Python 3.7.2 (default, Feb 14 2019, 16:12:42)
[Clang 10.0.0 (clang-1000.10.44.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> ast.parse('print "foo"')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/philsnow/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 35, in parse
return compile(source, filename, mode, PyCF_ONLY_AST)
File "<unknown>", line 1
print "foo"
^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print("foo")?
nothing changed in the text, but the AST is different. What should get committed to this hypothetical "git-ast"?
The answer probably lies in creating a different abstraction , different building blocks. In git we have bytes -> blobs -> trees -> commits, where the bytes of a given file get annotated with a file name (and mode bits etc etc, gross posix things) by the tree object.
In git-ast, maybe the abstraction is bytes -> lexemes (which annotates the bytes with a language version, and thus lexer version) -> tree? I haven't gone down the thought experiment that far.
- jlarocco 6y agoI suspect you're probably right, but that example isn't the greatest because Python 3 isn't source compatible with Python 2. The source code text itself isn't even backwards compatible in that case.
- philsnow 6y agoThat was maybe not a great example, pick any language change where a new keyword was introduced that wasn't previously a keyword or reserved word. I tried to make an example with `with`, because that was introduced between 2.4 and 2.5 IIRC, and 2.4 and 2.5 aren't nearly as far apart as 2.* and 3.*, but I didn't find an easy way to install a 2.4 python (it didn't compile in pyenv on my mac, and I looked at a couple docker images before giving up).
- Dylan16807 6y agoBut you're describing a benefit of storing an AST. Instead of the code suddenly changing meaning, you have an AST that says what the meaning should be. The worst case is that you export it with the old version of the converter, and then the code changes meaning. You have silent corruption. But if you stored raw text this would always happen. The good case is that you export it with the new version of the converter. It will notice that a variable name matches a keyword, and alert you to fix it.
- iso8859-1 6y agoSince even minor releases can introduce new syntax that previously failed to parse, this happens more than just on major version numbers. For example: async/await, formatting string literals, underscores in numeric literals, the list goes on and on. So you'll have to upgrade your formatter. And when you upgrade it, it may start formatting something differently than it previously did, even if that chunk had no new syntax.
- Dylan16807 6y agoUse one that won't change how it formats on a whim?