3 ms·

This looks very interesting! I hope that maybe this project can include a “refactoring parser”, that is, for each token in the parse tree, store where in the so

by goerz 7y ago

This looks very interesting! I hope that maybe this project can include a “refactoring parser”, that is, for each token in the parse tree, store where in the source code file that token originated from. Also, white space and comments have would have to be included, such that the round trip (source file -> AST -> source file) is possible without loss. I’ve long wanted to write some refactoring tools for modern Fortran, but not having such a parser is a big blocker—there’s only so much you can do with regexes.

jagger27 7y ago
Wouldn't writing an Antlr grammar handle this?
- goerz 7y ago
  Does Antlr support an annotated parse tree that would allow for round-tripping? I didn't see anything in the documentation, at a quick glance. Even then, formulating a grammar (especially one that also catches whitespace and comments) seems like a pretty daunting task.
  - jagger27 7y ago
    Catching whitespace and comments is no problem in Antlr[1]. I'm not exactly sure what you mean by round-tripping. In terms of how daunting it is, it's much easier than trying to use Flex/Bison or lex/yacc, that's for sure. 1: At the bottom of the grammar, there are rules to intentionally skip over whitespace and comments, but it could have just as easily captured them and parsed them further. https://github.com/antlr/grammars-v4/blob/master/c/C.g4 https://github.com/antlr/grammars-v4/blob/master/c/C.g4
    - goerz 7y ago
      What I mean by "round-tripping" is that I would have to be able to read a .f90 file into an AST structure, then write out the tree to a new file, and end up with the exact same file content as the original. This would be a prerequisite for a refactoring tool, which has to preserve comments and code formatting.
certik 7y ago
One of LFortran authors here. Yes, we already have an open issue for exactly this: https://gitlab.com/lfortran/lfortran/issues/42 https://gitlab.com/lfortran/lfortran/issues/42. As you can read there, it's actually not that easy as I first thought. But I think it's very much worth pursuing. For example, Python's AST is not round-trippable, and that forces people to write alternative parsers such as Parso (https://github.com/davidhalter/parso https://github.com/davidhalter/parso). ANTLR should allow to parse all the white space. But the issue is how to represent it in the AST, see the issue for more details. One could have an AST and a parser just for this round-trippable application, but that defeats the purpose. The goal should be to have this part of the compiler so that one can trust that it parses things correctly. Right now I am concentrating my efforts to finish gfortran compatibility (see the roadmap at https://lfortran.org https://lfortran.org) and to get first users. Once we get further along, we will tackle this problem. I am hopeful there might be a way to do this.

by goerz 7y ago

jagger27 7y ago

Wouldn't writing an Antlr grammar handle this?

goerz 7y ago
Does Antlr support an annotated parse tree that would allow for round-tripping? I didn't see anything in the documentation, at a quick glance. Even then, formulating a grammar (especially one that also catches whitespace and comments) seems like a pretty daunting task.
- jagger27 7y ago
  Catching whitespace and comments is no problem in Antlr[1]. I'm not exactly sure what you mean by round-tripping. In terms of how daunting it is, it's much easier than trying to use Flex/Bison or lex/yacc, that's for sure. 1: At the bottom of the grammar, there are rules to intentionally skip over whitespace and comments, but it could have just as easily captured them and parsed them further. https://github.com/antlr/grammars-v4/blob/master/c/C.g4 https://github.com/antlr/grammars-v4/blob/master/c/C.g4
  - goerz 7y ago
    What I mean by "round-tripping" is that I would have to be able to read a .f90 file into an AST structure, then write out the tree to a new file, and end up with the exact same file content as the original. This would be a prerequisite for a refactoring tool, which has to preserve comments and code formatting.

certik 7y ago

One of LFortran authors here. Yes, we already have an open issue for exactly this: https://gitlab.com/lfortran/lfortran/issues/42 https://gitlab.com/lfortran/lfortran/issues/42. As you can read there, it's actually not that easy as I first thought. But I think it's very much worth pursuing. For example, Python's AST is not round-trippable, and that forces people to write alternative parsers such as Parso (https://github.com/davidhalter/parso https://github.com/davidhalter/parso). ANTLR should allow to parse all the white space. But the issue is how to represent it in the AST, see the issue for more details. One could have an AST and a parser just for this round-trippable application, but that defeats the purpose. The goal should be to have this part of the compiler so that one can trust that it parses things correctly. Right now I am concentrating my efforts to finish gfortran compatibility (see the roadmap at https://lfortran.org https://lfortran.org) and to get first users. Once we get further along, we will tackle this problem. I am hopeful there might be a way to do this.