7 ms·
Using the line/column information in Python's ast objects, you can sort of, almost recreate the concrete syntax, and I've tried to do so in my toy project https
by rav 10y ago
Using the line/column information in Python's ast objects, you can sort of, almost recreate the concrete syntax, and I've tried to do so in my toy project https://github.com/Mortal/fstrings https://github.com/Mortal/fstrings which converts old %-style formatting to the new Python 3.6 f-string syntax. It's not feature complete, but it mostly works.
- DasIch 10y agoThe lib2to3 library provides concrete syntax trees.
- chubot 10y agoYes I just mentioned lib2to3 in the post, that's probably the closest example. (There are also some Clang-based tools to convert C++ 03 to 11 to 14, etc.) Although I believe that you need different terminology. A concrete syntax tree doesn't necessarily represent comments and whitespace. It represents the derivation of the string from the grammar. These are totally different things and I give an example in the article. The CST is kind of wasteful and I'm not sure why you would use it other than not wanting to write semantic actions to prune the tree on the fly. If anyone has insight into this, please let me know.
- FlorianRappl 10y agoWhile line/column works nicely if no modification was done on the AST, it works pretty bad with modification. Furthermore, you can't do cosmetic (e.g., indentation) changes on the AST that will reflect back on the source code (preserving user, i.e., non-modified, style decisions). The Microsoft example (thinking of Roslyn) offers a great lossless syntax tree written with immutable data structures.
- icebraining 10y agoThere's redbaron, which provides a full syntax tree, and is designed for making such changes: https://github.com/PyCQA/redbaron https://github.com/PyCQA/redbaron