5 ms·

The file containing the code is text. What AST do you mean? Cog doesn't have to understand anything about the structure of the containing file, so it can be

by nedbat 8y ago

The file containing the code is text. What AST do you mean? Cog doesn't have to understand anything about the structure of the containing file, so it can be used on any text file.

snazz 8y ago
Processing the language’s abstract syntax tree is much less error-prone, for one. C macros (and any other text based macro system, including Cog) can be harder to debug than an AST version. While you could implement something complex, like list comprehensions in Cog, it would be a lot more complex and buggy than the AST-modifying equivalent[0]. That’s probably not what the author (you?) had in mind for Cog, but it is effectively a macro system so tying into the AST could be a big advantage. The only nice part about the text version is that it doesn’t presuppose a host language. [0]:https://stackoverflow.com/questions/267862/what-makes-lisp-macros-so-special https://stackoverflow.com/questions/267862/what-makes-lisp-m...
- ironmagma 8y ago
  Less error-prone in one sense, more error-prone in another. Any time you have an AST integration, you will run into versioning and integration issues. Consider for example a modern ES6 stack with Babel -- will all your third party tooling recognize the latest syntaxes recognized by Babel? And if so, will it all output AST to text in the correct way? Probably not. Same goes for versioning in languages like Python 2 vs 3 where the AST is only slightly different. It's much simpler for a tool like this to be "dumb" -- leave the correct syntax to the human, since it's a lot easier for a human to deal with on a one-off basis than to have a group of developers writing and maintaining dozens of AST parsers and code generators.
  - xg15 8y ago
    Given the number of code injection vulnerabilities and escaping confusions we see, I disagree with the assertion that humans are good at understanding code in the same way that parsers are. Also, any modern IDE already needs to parse the AST anyway to provide any half-decent inspection features. The grammars/specs of most popular languages are also readily available and well-maintained. On the contrary, if you add search/replace style macro processing, you actually make the code more difficult to formally analyze because, then not even an IDE could build a meaningful AST without actually expanding the macros. (which in this case would mean executing arbitrary python code)
    - wahern 8y ago
      > Also, any modern IDE already needs to parse the AST anyway to provide any half-decent inspection features. The grammars/specs of most popular languages are also readily available and well-maintained. But do those IDEs actually execute the in-language generators? Which AST are they reporting, the preprocessed or postprocessed? I've heavily used M4 with C code before and what I liked was the ability to see and run tools against the postprocessed code. Textual replacement can be thought of as a worse is better approach, a principle behind many of the systems that people enjoy and even find to be elegant (at least as long as they don't look too closely).
      - rightbyte 8y ago
        The point with cog is that you will have the output in the file also. The ide doesn't need to understand the generator. I like Cog. It is usefull for a set of problems. Eg. writing big enums for cpp with enum-to-string and back functions etc.
xg15 8y ago
I can understand the priority of making the tool language-neutral, but by making it agnostic of the underyling structure, this can also cause lots of "code injection" like problems where the generated code behaves in many unintuitive ways. See e.g. https://stuff.mit.edu/afs/athena/project/rhel-doc/3/rhel-cpp-en-3/macro-pitfalls.html https://stuff.mit.edu/afs/athena/project/rhel-doc/3/rhel-cpp... for examples.

by nedbat 8y ago

The file containing the code is text. What AST do you mean? Cog doesn't have to understand anything about the structure of the containing file, so it can be used on any text file.

snazz 8y ago

Processing the language’s abstract syntax tree is much less error-prone, for one. C macros (and any other text based macro system, including Cog) can be harder to debug than an AST version. While you could implement something complex, like list comprehensions in Cog, it would be a lot more complex and buggy than the AST-modifying equivalent[0]. That’s probably not what the author (you?) had in mind for Cog, but it is effectively a macro system so tying into the AST could be a big advantage. The only nice part about the text version is that it doesn’t presuppose a host language. [0]:https://stackoverflow.com/questions/267862/what-makes-lisp-macros-so-special https://stackoverflow.com/questions/267862/what-makes-lisp-m...

ironmagma 8y ago
Less error-prone in one sense, more error-prone in another. Any time you have an AST integration, you will run into versioning and integration issues. Consider for example a modern ES6 stack with Babel -- will all your third party tooling recognize the latest syntaxes recognized by Babel? And if so, will it all output AST to text in the correct way? Probably not. Same goes for versioning in languages like Python 2 vs 3 where the AST is only slightly different. It's much simpler for a tool like this to be "dumb" -- leave the correct syntax to the human, since it's a lot easier for a human to deal with on a one-off basis than to have a group of developers writing and maintaining dozens of AST parsers and code generators.
- xg15 8y ago
  Given the number of code injection vulnerabilities and escaping confusions we see, I disagree with the assertion that humans are good at understanding code in the same way that parsers are. Also, any modern IDE already needs to parse the AST anyway to provide any half-decent inspection features. The grammars/specs of most popular languages are also readily available and well-maintained. On the contrary, if you add search/replace style macro processing, you actually make the code more difficult to formally analyze because, then not even an IDE could build a meaningful AST without actually expanding the macros. (which in this case would mean executing arbitrary python code)
  - wahern 8y ago
    > Also, any modern IDE already needs to parse the AST anyway to provide any half-decent inspection features. The grammars/specs of most popular languages are also readily available and well-maintained. But do those IDEs actually execute the in-language generators? Which AST are they reporting, the preprocessed or postprocessed? I've heavily used M4 with C code before and what I liked was the ability to see and run tools against the postprocessed code. Textual replacement can be thought of as a worse is better approach, a principle behind many of the systems that people enjoy and even find to be elegant (at least as long as they don't look too closely).
    - rightbyte 8y ago
      The point with cog is that you will have the output in the file also. The ide doesn't need to understand the generator. I like Cog. It is usefull for a set of problems. Eg. writing big enums for cpp with enum-to-string and back functions etc.

xg15 8y ago

I can understand the priority of making the tool language-neutral, but by making it agnostic of the underyling structure, this can also cause lots of "code injection" like problems where the generated code behaves in many unintuitive ways. See e.g. https://stuff.mit.edu/afs/athena/project/rhel-doc/3/rhel-cpp-en-3/macro-pitfalls.html https://stuff.mit.edu/afs/athena/project/rhel-doc/3/rhel-cpp... for examples.