5 ms·

> get the AST (a data structure AFAIK), manipulate it in a sane fashion in Python, compile it, return the result. No string munching eval require I use both Li

> get the AST (a data structure AFAIK), manipulate it in a sane fashion in Python, compile it, return the result. No string munching eval require I use both Lisps and Python, and especially after learning to use Lisps I've wanted to do this in Python, so I looked for the ability to do what you describe here. While it is technically possible to do this (like most things) in Python, it is quite the effort and nowhere near ergonomic. Among all the cases where I could really use this, I found it worth the effort to instead: 1. Manipulate & generate code statically (from ASTs), as a build-time / dev-time step, using redbaron etc.; or 2. Just use a Lisp (specifically, Hy). Both methods involved writing quite a lot of code. In the former: to load, parse, match, and transform ASTs; in the latter: to port existing code (the code I originally wanted to manipulate) from Python to Hy. ---- The reason for all this is plainly the fact that Lisps are designed for this, while Python is not. Before learning Lisp, I used to wonder what could be so different in the language itself that could possibly make it that much easier. Now I use a Lisp-in-Python in production precisely because of this.

mark_l_watson 7y ago
Thank you, interesting comments, especially switching over to the Hy language for production. I got very excited about Hy, then an update took away a built in “let” and that turned me off a bit. Are there any good directions for patching in the old code supporting “let”?
marmaduke 7y ago
This is true, but I wanted to correct factual errors in the article. If you or your team are Lisp users then Hy or Clojure would be good choices, but if you are just trying to create a transformation for existing Python codebases, the decorator approach is a good shoehorn. On the other hand, you can also interpret the difficulty of AST transforms as a signal from the language designers (had van Rossum really never heard of Lisp?) that they hurt readability and their use should require a little ceremony.
- pritambaral 7y ago
  I'm sorry, but I don't agree at all. > if you are just trying to create a transformation for existing Python codebases, the decorator approach is a good shoehorn. For transformations that need to modify the code, decorators are insufficient. Decorators have their value, and I have built good interfaces using decorators, which would have been rather difficult without them, but all of that was because decorators help _wrap_, not _transform_ code. When I had a need to transform code, I looked for the right tool, and the takeaways of my searches are mentioned in my original comment. > you can also interpret the difficulty of AST transforms as a signal from the language designers (had van Rossum really never heard of Lisp?) that they hurt readability and their use should require a little ceremony. Nobody's claiming Guido had never heard of Lisp, or any non-lispy-language designer for that matter. Language designers make languages for varied reasons. Guido's reasons actually happen to be documented, and the only reason Python isn't homoiconic is because Guido didn't have a need for it, not because he explicitly wanted AST transforms to "require a little ceremony". In fact, Python AST transforms do not merely "require a little ceremony". They are actually quite cumbersome. The standard library actually provides no help at all in transforms, making it necessary for libraries like Rope, RedBaron, and Astor to be written.
ken 7y ago
> While it is technically possible to do this (like most things) in Python, it is quite the effort and nowhere near ergonomic. Indeed! Peter Norvig once made the same observation [1]: > Python does have access to the abstract syntax tree of programs, but this is not for the faint of heart. On the plus side, the modules are easy to understand, and with five minutes and five lines of code I was able to get this: >>> parse("2 + 2") ['eval_input', ['testlist', ['test', ['and_test', ['not_test', ['comparison', ['expr', ['xor_expr', ['and_expr', ['shift_expr', ['arith_expr', ['term', ['factor', ['power', ['atom', [2, '2']]]]], [14, '+'], ['term', ['factor', ['power', ['atom', [2, '2']]]]]]]]]]]]]]], [4, ''], [0, '']] > This was rather a disapointment to me. The Lisp parse of the equivalent expression is (+ 2 2). It seems that only a real expert would want to manipulate Python parse trees, whereas Lisp parse trees are simple for anyone to use. The essential property of a Lisp, to me, is that the syntax is the AST. For any language which does not have this property, providing DOM-like access to the AST is neat but simply not practical for the user. [1]: https://norvig.com/python-lisp.html https://norvig.com/python-lisp.html

mark_l_watson 7y ago

Thank you, interesting comments, especially switching over to the Hy language for production. I got very excited about Hy, then an update took away a built in “let” and that turned me off a bit. Are there any good directions for patching in the old code supporting “let”?

marmaduke 7y ago

This is true, but I wanted to correct factual errors in the article. If you or your team are Lisp users then Hy or Clojure would be good choices, but if you are just trying to create a transformation for existing Python codebases, the decorator approach is a good shoehorn. On the other hand, you can also interpret the difficulty of AST transforms as a signal from the language designers (had van Rossum really never heard of Lisp?) that they hurt readability and their use should require a little ceremony.

pritambaral 7y ago
I'm sorry, but I don't agree at all. > if you are just trying to create a transformation for existing Python codebases, the decorator approach is a good shoehorn. For transformations that need to modify the code, decorators are insufficient. Decorators have their value, and I have built good interfaces using decorators, which would have been rather difficult without them, but all of that was because decorators help _wrap_, not _transform_ code. When I had a need to transform code, I looked for the right tool, and the takeaways of my searches are mentioned in my original comment. > you can also interpret the difficulty of AST transforms as a signal from the language designers (had van Rossum really never heard of Lisp?) that they hurt readability and their use should require a little ceremony. Nobody's claiming Guido had never heard of Lisp, or any non-lispy-language designer for that matter. Language designers make languages for varied reasons. Guido's reasons actually happen to be documented, and the only reason Python isn't homoiconic is because Guido didn't have a need for it, not because he explicitly wanted AST transforms to "require a little ceremony". In fact, Python AST transforms do not merely "require a little ceremony". They are actually quite cumbersome. The standard library actually provides no help at all in transforms, making it necessary for libraries like Rope, RedBaron, and Astor to be written.

ken 7y ago

> While it is technically possible to do this (like most things) in Python, it is quite the effort and nowhere near ergonomic. Indeed! Peter Norvig once made the same observation [1]: > Python does have access to the abstract syntax tree of programs, but this is not for the faint of heart. On the plus side, the modules are easy to understand, and with five minutes and five lines of code I was able to get this: >>> parse("2 + 2") ['eval_input', ['testlist', ['test', ['and_test', ['not_test', ['comparison', ['expr', ['xor_expr', ['and_expr', ['shift_expr', ['arith_expr', ['term', ['factor', ['power', ['atom', [2, '2']]]]], [14, '+'], ['term', ['factor', ['power', ['atom', [2, '2']]]]]]]]]]]]]]], [4, ''], [0, '']] > This was rather a disapointment to me. The Lisp parse of the equivalent expression is (+ 2 2). It seems that only a real expert would want to manipulate Python parse trees, whereas Lisp parse trees are simple for anyone to use. The essential property of a Lisp, to me, is that the syntax is the AST. For any language which does not have this property, providing DOM-like access to the AST is neat but simply not practical for the user. [1]: https://norvig.com/python-lisp.html https://norvig.com/python-lisp.html