7 ms·

Code autoformatting is amazing and I never want to go back; even though I strenuously disagree with the particular formatting my team/company has chosen for pyt

Code autoformatting is amazing and I never want to go back; even though I strenuously disagree with the particular formatting my team/company has chosen for python, I'm so relieved to never ever have a style discussion in PRs that I don't care about double quotes vs single quotes anymore. BUT I'm tired of seeing diff chunks on github where 95% of the chunk is because black/gofmt decided to change the formatting / indentation / grouping / whatever. I want language-agnostic, AST-aware diffing, that de-emphasizes the part of a chunk that is irrelevant-in-the-world-of-auto-matic-code-formatting.

dllthomas 6y ago
Hopefully you're already aware but just in case: many tools (including git and github) have an option somewhere to hide whitespace changes. In some contexts this can help a lot, though it certainly doesn't get us all the way there.
nxrabl 6y ago
Append “?w=1” to the end of a Github PR URL, it’s like passing “-w” to git. (Not that that addresses all of your comment, I just wanted to share the tip. Agreed that smarter diffing would be really nice.)
- u801e 6y ago
  git diff has a --diff-algorithm parameter that has several options to choose from. I don't know whether Github supports that via url parameters though.
kristo 6y ago
Why don’t we just check ASTs in to git instead of text files ?
- ssivark 6y ago
  I think this is the key question, and doing this will enable numerous interface improvements for people working with software (Eg: separation of content and presents means that everybody can use their favorite view — three-space-tabs, extra-long lines, whatever). The challenge is that version control on text files (list of lines) is fairly straight forward and language agnostic, while OTOH the VCS would have to run after the language parser processes on the text file, and then be able to interface with the AST generated by the parser (for multiple languages). I presume this would become easy if different languages have the same parsing setup, or we have language/project-specific VCS tools. While the answers are not obvious (to me, at least) this definitely seems like the right direction to move in, instead of getting hung up on plain text and formatting issues.
  - nyanpasu64 6y ago
    Storing an AST instead of text might be a way to eliminate parsing ambiguity of angle brackets for generic arguments (as opposed to comparisons). It might still be visually ambiguous, but I don't think it's a problem in practice (if the display forces whitespace around comparisons)?
- i_dursun 6y ago
  I believe unison does something along those lines: https://www.unisonweb.org/ https://www.unisonweb.org/
- tlrobinson 6y ago
  There are some cases where formatters like Prettier don't have one canonical representation for an AST, in particular behavior around line breaks (multiple empty lines are allowed, a line break after an "{" opening brace in an object literal will force it to span multiple lines, etc) But yeah, otherwise everyone could just have their own preferred formatter/settings and we could commit whatever representation is most optimal for diffing to version control (e.x. multi-line object literals, trailing commas in arrays/args, etc). You could use something more compact like a binary AST but for compatibility with "legacy" tools that deal with text I think it would be better to use a more verbose text-based representation.
- pmiller2 6y ago
  Because that requires either compiler support, or a plug-in to be written for each language and language version. But, I guess you can always fall back to text files in the worst case.
- cjohansson 6y ago
  Sounds like a great idea, or binary representation of lexer tokens, the AST is larger in size than the lexer tokens and the AST can be easily generated from the tokens
- jyounker 6y ago
  I believe that's called "lisp".
  - Firadeoclus 6y ago
    Which lisps don't use text files for source code?
- philsnow 6y ago
  I have thought about this a bit in the past, and I think the answer is that the AST for a given blob of text can be different depending on which version of the language you interpret it with. consider 'print "foo"' in python 2.7: Python 2.7.15 (default, Feb 14 2019, 16:26:56) [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.10.44.4)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import ast >>> ast.dump(ast.parse('print "foo"')) "Module(body=[Print(dest=None, values=[Str(s='foo')], nl=True)])" but in python 3.7: Python 3.7.2 (default, Feb 14 2019, 16:12:42) [Clang 10.0.0 (clang-1000.10.44.4)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import ast >>> ast.parse('print "foo"') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/philsnow/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 35, in parse return compile(source, filename, mode, PyCF_ONLY_AST) File "<unknown>", line 1 print "foo" ^ SyntaxError: Missing parentheses in call to 'print'. Did you mean print("foo")? nothing changed in the text, but the AST is different. What should get committed to this hypothetical "git-ast"? The answer probably lies in creating a different abstraction , different building blocks. In git we have bytes -> blobs -> trees -> commits, where the bytes of a given file get annotated with a file name (and mode bits etc etc, gross posix things) by the tree object. In git-ast, maybe the abstraction is bytes -> lexemes (which annotates the bytes with a language version, and thus lexer version) -> tree? I haven't gone down the thought experiment that far.
  - jlarocco 6y ago
    I suspect you're probably right, but that example isn't the greatest because Python 3 isn't source compatible with Python 2. The source code text itself isn't even backwards compatible in that case.
    - philsnow 6y ago
      That was maybe not a great example, pick any language change where a new keyword was introduced that wasn't previously a keyword or reserved word. I tried to make an example with `with`, because that was introduced between 2.4 and 2.5 IIRC, and 2.4 and 2.5 aren't nearly as far apart as 2.* and 3.*, but I didn't find an easy way to install a 2.4 python (it didn't compile in pyenv on my mac, and I looked at a couple docker images before giving up).
- mook 6y ago
  I kind of like the ability to have comments, though. The tools I've worked with have issues round-tripping them though ASTs.
tlrobinson 6y ago
This is basically solved by pinning the formatter to a specific version in your project's dependencies, and enforcing its use in continuous integration. Of course that doesn't help if you're setting up the formatter for the first time, or want to upgrade to a new version. Some tools like GitHub's diff do try to deemphasize whitespace but it's pretty limited.
- l0b0 6y ago
  In easy steps (Python/Git specific): 1. Use a tracked directory for Git hooks: `git config --local core.hooksPath 'git-hooks'`. 2. Create format.bash and lint.bash. 2.1. Run `black` and `isort` in format.bash. 2.2. Run `flake8`, `mypy`, `black` and `isort` in lint.bash. 2.3. Run lint.bash in the pre-commit hook. 2.4. By default, lint/format only files which are different from origin/master to save time: `git diff --diff-filter=d --name-only -z origin/master "[asterisk].py"`. 3. In CI, run lint.bash with `--all` or some other keyword to verify formatting on every file: `git ls-files -z -- "[asterisk].py"`. After more than a year with this setup I absolutely adore it.
  - tlrobinson 6y ago
    Even better, IMHO, is just setting your editor to format on save. That way you can just write ugly code and hit save to get it formatted. I've found git hooks to be more trouble than they're worth. But as long as it's enforced in CI you can leave it up to individual team members how they want to do it.
deadbunny 6y ago
The frustration is real in a PR situation when you have your editor set to run a formatter on save (same thing with stripping whitespace) but if you have enough control of the project and want to switch to using a formatter then git can help you there. You can do a format run over the whole codebase then ignore the commit in blame. https://www.moxio.com/blog/43/ignoring-bulk-change-commits-with-git-blame https://www.moxio.com/blog/43/ignoring-bulk-change-commits-w...

dllthomas 6y ago

Hopefully you're already aware but just in case: many tools (including git and github) have an option somewhere to hide whitespace changes. In some contexts this can help a lot, though it certainly doesn't get us all the way there.

nxrabl 6y ago

Append “?w=1” to the end of a Github PR URL, it’s like passing “-w” to git. (Not that that addresses all of your comment, I just wanted to share the tip. Agreed that smarter diffing would be really nice.)

u801e 6y ago
git diff has a --diff-algorithm parameter that has several options to choose from. I don't know whether Github supports that via url parameters though.

kristo 6y ago

Why don’t we just check ASTs in to git instead of text files ?

ssivark 6y ago
I think this is the key question, and doing this will enable numerous interface improvements for people working with software (Eg: separation of content and presents means that everybody can use their favorite view — three-space-tabs, extra-long lines, whatever). The challenge is that version control on text files (list of lines) is fairly straight forward and language agnostic, while OTOH the VCS would have to run after the language parser processes on the text file, and then be able to interface with the AST generated by the parser (for multiple languages). I presume this would become easy if different languages have the same parsing setup, or we have language/project-specific VCS tools. While the answers are not obvious (to me, at least) this definitely seems like the right direction to move in, instead of getting hung up on plain text and formatting issues.
- nyanpasu64 6y ago
  Storing an AST instead of text might be a way to eliminate parsing ambiguity of angle brackets for generic arguments (as opposed to comparisons). It might still be visually ambiguous, but I don't think it's a problem in practice (if the display forces whitespace around comparisons)?
i_dursun 6y ago
I believe unison does something along those lines: https://www.unisonweb.org/ https://www.unisonweb.org/
tlrobinson 6y ago
There are some cases where formatters like Prettier don't have one canonical representation for an AST, in particular behavior around line breaks (multiple empty lines are allowed, a line break after an "{" opening brace in an object literal will force it to span multiple lines, etc) But yeah, otherwise everyone could just have their own preferred formatter/settings and we could commit whatever representation is most optimal for diffing to version control (e.x. multi-line object literals, trailing commas in arrays/args, etc). You could use something more compact like a binary AST but for compatibility with "legacy" tools that deal with text I think it would be better to use a more verbose text-based representation.
pmiller2 6y ago
Because that requires either compiler support, or a plug-in to be written for each language and language version. But, I guess you can always fall back to text files in the worst case.
cjohansson 6y ago
Sounds like a great idea, or binary representation of lexer tokens, the AST is larger in size than the lexer tokens and the AST can be easily generated from the tokens
jyounker 6y ago
I believe that's called "lisp".
- Firadeoclus 6y ago
  Which lisps don't use text files for source code?
philsnow 6y ago
I have thought about this a bit in the past, and I think the answer is that the AST for a given blob of text can be different depending on which version of the language you interpret it with. consider 'print "foo"' in python 2.7: Python 2.7.15 (default, Feb 14 2019, 16:26:56) [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.10.44.4)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import ast >>> ast.dump(ast.parse('print "foo"')) "Module(body=[Print(dest=None, values=[Str(s='foo')], nl=True)])" but in python 3.7: Python 3.7.2 (default, Feb 14 2019, 16:12:42) [Clang 10.0.0 (clang-1000.10.44.4)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import ast >>> ast.parse('print "foo"') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/philsnow/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 35, in parse return compile(source, filename, mode, PyCF_ONLY_AST) File "<unknown>", line 1 print "foo" ^ SyntaxError: Missing parentheses in call to 'print'. Did you mean print("foo")? nothing changed in the text, but the AST is different. What should get committed to this hypothetical "git-ast"? The answer probably lies in creating a different abstraction , different building blocks. In git we have bytes -> blobs -> trees -> commits, where the bytes of a given file get annotated with a file name (and mode bits etc etc, gross posix things) by the tree object. In git-ast, maybe the abstraction is bytes -> lexemes (which annotates the bytes with a language version, and thus lexer version) -> tree? I haven't gone down the thought experiment that far.
- jlarocco 6y ago
  I suspect you're probably right, but that example isn't the greatest because Python 3 isn't source compatible with Python 2. The source code text itself isn't even backwards compatible in that case.
  - philsnow 6y ago
    That was maybe not a great example, pick any language change where a new keyword was introduced that wasn't previously a keyword or reserved word. I tried to make an example with `with`, because that was introduced between 2.4 and 2.5 IIRC, and 2.4 and 2.5 aren't nearly as far apart as 2.* and 3.*, but I didn't find an easy way to install a 2.4 python (it didn't compile in pyenv on my mac, and I looked at a couple docker images before giving up).
mook 6y ago
I kind of like the ability to have comments, though. The tools I've worked with have issues round-tripping them though ASTs.

tlrobinson 6y ago

This is basically solved by pinning the formatter to a specific version in your project's dependencies, and enforcing its use in continuous integration. Of course that doesn't help if you're setting up the formatter for the first time, or want to upgrade to a new version. Some tools like GitHub's diff do try to deemphasize whitespace but it's pretty limited.

l0b0 6y ago
In easy steps (Python/Git specific): 1. Use a tracked directory for Git hooks: `git config --local core.hooksPath 'git-hooks'`. 2. Create format.bash and lint.bash. 2.1. Run `black` and `isort` in format.bash. 2.2. Run `flake8`, `mypy`, `black` and `isort` in lint.bash. 2.3. Run lint.bash in the pre-commit hook. 2.4. By default, lint/format only files which are different from origin/master to save time: `git diff --diff-filter=d --name-only -z origin/master "[asterisk].py"`. 3. In CI, run lint.bash with `--all` or some other keyword to verify formatting on every file: `git ls-files -z -- "[asterisk].py"`. After more than a year with this setup I absolutely adore it.
- tlrobinson 6y ago
  Even better, IMHO, is just setting your editor to format on save. That way you can just write ugly code and hit save to get it formatted. I've found git hooks to be more trouble than they're worth. But as long as it's enforced in CI you can leave it up to individual team members how they want to do it.

deadbunny 6y ago

The frustration is real in a PR situation when you have your editor set to run a formatter on save (same thing with stripping whitespace) but if you have enough control of the project and want to switch to using a formatter then git can help you there. You can do a format run over the whole codebase then ignore the commit in blame. https://www.moxio.com/blog/43/ignoring-bulk-change-commits-with-git-blame https://www.moxio.com/blog/43/ignoring-bulk-change-commits-w...