5 ms·

> homoiconicity, meaning code can be represented as a data structure inside the same language > generating strings and passing them into eval like for example

> homoiconicity, meaning code can be represented as a data structure inside the same language > generating strings and passing them into eval like for example in Python Python has become a pinata for language enthusiasts? Python lets you pin a decorator on a function, get the AST (a data structure AFAIK), manipulate it in a sane fashion in Python, compile it, return the result. No string munching eval require I'm all for Lisp but this just isn't convincing

vaylian 7y ago
> get the AST (a data structure AFAIK) how?
- typon 7y ago
  https://docs.python.org/3/library/ast.html https://docs.python.org/3/library/ast.html
- masklinn 7y ago
  ast.parse(inspect.getsource(func)) doesn't work in the python shell though. You could also decompile the function's code object.
masklinn 7y ago
TBF I don't know how common it is as Python's AST API is pretty gnarly (not to mention really hard to work with / debug as it will usually segfault if you look at it funny). And IIRC the error messages coming from there are significantly worse than those coming from source. Third-party packages do make that somewhat better, but still not great. IIRC Jinja compiles templates to Python source code then compiles that to bytecode using the relevant builtins because it's simpler & more reliable than doing straight AST manipulations.
pritambaral 7y ago
> get the AST (a data structure AFAIK), manipulate it in a sane fashion in Python, compile it, return the result. No string munching eval require I use both Lisps and Python, and especially after learning to use Lisps I've wanted to do this in Python, so I looked for the ability to do what you describe here. While it is technically possible to do this (like most things) in Python, it is quite the effort and nowhere near ergonomic. Among all the cases where I could really use this, I found it worth the effort to instead: 1. Manipulate & generate code statically (from ASTs), as a build-time / dev-time step, using redbaron etc.; or 2. Just use a Lisp (specifically, Hy). Both methods involved writing quite a lot of code. In the former: to load, parse, match, and transform ASTs; in the latter: to port existing code (the code I originally wanted to manipulate) from Python to Hy. ---- The reason for all this is plainly the fact that Lisps are designed for this, while Python is not. Before learning Lisp, I used to wonder what could be so different in the language itself that could possibly make it that much easier. Now I use a Lisp-in-Python in production precisely because of this.
- mark_l_watson 7y ago
  Thank you, interesting comments, especially switching over to the Hy language for production. I got very excited about Hy, then an update took away a built in “let” and that turned me off a bit. Are there any good directions for patching in the old code supporting “let”?
- marmaduke 7y ago
  This is true, but I wanted to correct factual errors in the article. If you or your team are Lisp users then Hy or Clojure would be good choices, but if you are just trying to create a transformation for existing Python codebases, the decorator approach is a good shoehorn. On the other hand, you can also interpret the difficulty of AST transforms as a signal from the language designers (had van Rossum really never heard of Lisp?) that they hurt readability and their use should require a little ceremony.
  - pritambaral 7y ago
    I'm sorry, but I don't agree at all. > if you are just trying to create a transformation for existing Python codebases, the decorator approach is a good shoehorn. For transformations that need to modify the code, decorators are insufficient. Decorators have their value, and I have built good interfaces using decorators, which would have been rather difficult without them, but all of that was because decorators help _wrap_, not _transform_ code. When I had a need to transform code, I looked for the right tool, and the takeaways of my searches are mentioned in my original comment. > you can also interpret the difficulty of AST transforms as a signal from the language designers (had van Rossum really never heard of Lisp?) that they hurt readability and their use should require a little ceremony. Nobody's claiming Guido had never heard of Lisp, or any non-lispy-language designer for that matter. Language designers make languages for varied reasons. Guido's reasons actually happen to be documented, and the only reason Python isn't homoiconic is because Guido didn't have a need for it, not because he explicitly wanted AST transforms to "require a little ceremony". In fact, Python AST transforms do not merely "require a little ceremony". They are actually quite cumbersome. The standard library actually provides no help at all in transforms, making it necessary for libraries like Rope, RedBaron, and Astor to be written.
ertucetin 7y ago
Python is not as flexible as Lisp in a manner of AST manipulation
- smitty1e 7y ago
  Python's approach seems to be: expose functionality like the AST for the 31337 ninjas that really want to go there, while maintaining intense focus on producing a KISS tool for the other 95% of its audience.
  - jacobobryant 7y ago
    A lot of (most?) of the benefit of macros comes not from writing them yourself, but from using macros that others have written. Even if only 5% of the people are writing macros, that 5% likely includes many library authors. Making it harder for them impacts their downstream users too, even if those people aren't actually writing macros. IMO the macros in Clojure make it easier for library authors to produce KISS, high-level interfaces for the rest of us.
  - TeMPOraL 7y ago
    This ultimately impacts code quality negatively. Macros aren't a tool "for the 31337 ninjas", not more than class inheritance or generic types, or hell, even procedures. Macros solve a particular class of problems - they eliminate bloat and boilerplate, allowing you to express your code closer to the way you think about it. Much like classes can be used as data transfer objects or to build out a large inheritance tree, macros can be used tactically - to DRY up things you otherwise couldn't, or introduce some readability improvement - or strategically, to lift the code up another layer of abstraction, or create DSLs that let you write your code much closer to the problem domain. Because Lisp supports them in an ergonomic way, macros are not seen as something special, but just another tool in the toolbox.
lenkite 7y ago
You might enjoy this talk- Live Coding a Compiler using Clojure at https://youtu.be/_lMltUNB6Yw?list=PLhYmIiHOMWoEgJEvgkmUe8D0agxy_T2vR https://youtu.be/_lMltUNB6Yw?list=PLhYmIiHOMWoEgJEvgkmUe8D0a...
kopos 7y ago
A python decorator is just a syntactic sugar for : f = decorator(f) Afaik, the decorator deals with the internal function as a black box. But with lisp being a lisp, the macro writer can choose the level of blackboxiness they want to deal with. Note: I love both Python and Clojure and use them extensively.
- dangerbird2 7y ago
  As others have hinted at, higher-order functions like decorators can inspect and manipulate the internals of a "black box" function object with the ast module[1]. This would be comparable to a lisp function taking a quoted form, expanding and evaling it at runtime as an ad-hoc macro funcitonality as is common in more bare-bones lisp implementations. Aside from the ease of manipulating quoted forms (i.e. asts) in Lisp and Clojure, a big part of the power of defmacro and kin is that Lisps distinguish (depending on implementation) between read-time, macroexpand-time, compile time, and eval time, giving the macro author control over when and where the macro is executed as well as how. As an example, this makes things like lazy evaluation (without requiring users to explicitly "thunk" values into functions) trivial to implement in lisp [1] https://docs.python.org/3/library/ast.html https://docs.python.org/3/library/ast.html
  - kopos 7y ago
    Thank you, that was very informative. I do remember that Python had the ast module for a rules engine implementation, but I felt it was too cumbersome.. Thank you.
duckerude 7y ago
While you can manipulate Python's AST, you can't do it in a way that looks like adding new syntax. In Clojure you can write a list comprehension as `(for [x (range 10)] x)`. In Python you can write it as `[x for x in range(10)]`. Imagine these didn't exist, and you wanted to implement them yourself. In Clojure this is possible. Its list comprehensions are implemented as a Clojure macro: https://github.com/clojure/clojure/blob/b98ba84/src/clj/clojure/core.clj#L4590 https://github.com/clojure/clojure/blob/b98ba84/src/clj/cloj... In Python there are several problems. The first is that you can't easily pull the decorator trick on arbitrary expressions. Not even with a lambda function, at least without doing a lot of work - inspect.getsource doesn't handle those properly. But let's lower the bar, and replicate `l = [x for x in range(10)]` instead. You can do that with a decorator and an ordinary function. Then the problem becomes that you can only use existing syntax in the function body. Syntax that already means something else. If you put an ordinary list comprehension in the function body then you get a syntax error when the file is parsed, long before the decorator has a chance to do anything. So you have to make something up yourself. You might end up with something like this: @list_comp def l(): x: range(10) x Or, of course, this: l = list(map(lambda x: x, range(10))) Neither of these are as convenient or readable as real list comprehensions. But if you want those, you're at the whims of the language. You can't just add them yourself. I don't know how big of a deal this actually is. I've never used Clojure, and I've barely used other Lisps. Expressing what I want in Python is easy enough in practice (but maybe I just don't know any better?). But there's certainly something significant there that Python can't do.
- zcid 7y ago
  I heard somebody make a similar argument one time in regards to Common Lisp. He said that with Java you would have to wait several years sometimes to get a feature (assuming the language developers actually cared about the feature), but with Lisp, you could just implement the feature yourself. I've always felt this is one of the strongest arguments for using a lisp-style language. Edit: this is possible too in languages like Elixer and Ruby using macros, but eventually the readability suffers and debugging complex macros in those languages can be a nightmare.
  - deleted 7y ago
ken 7y ago
Python has become one of the most popular languages. I think they're just using it as an example of "here's a language you probably know which doesn't support this" -- or, as you note, kind of supports it now but makes it a real pain because it's not native to the syntax.
amelius 7y ago
Python's biggest problem wrt functional programming: multiline inline lambdas are not possible.
didibus 7y ago
It's more that code and data have the same representation. For example, where is the Json of Python? Where is the Python object notation which is both how you represent Python Objects in code as well as when serialized to data over a wire? [1 2 3] is both the serialized form of a vector as well as valid code to create a vector of itself. Now in JS, only object data is homoiconic. In Lisps, even code expressions are homoiconic, thus (+ 1 2) is the serialized form of the list of function + and args 1 and 2, as well as valid code to add 1 and 2 together. You do not need macros for homoiconicity. Macros is not what makes a language homoiconic. But once your language is homoiconic, it is very easy to add macros to it, and macros become much simpler to write. Which is why Lisps have macros as well.
iLemming 7y ago
> I'm all for Lisp but this just isn't convincing Nothing will convince you until you give it a heartfelt try. The joy from using Clojure is a real thing. Unfortunately, there are not too many languages out there you can say the same thing about (including Python). Clojure is not absolutely frustration-free PL, but it maintains a good balance between "fun" and "profit".

vaylian 7y ago

> get the AST (a data structure AFAIK) how?

typon 7y ago
https://docs.python.org/3/library/ast.html https://docs.python.org/3/library/ast.html
masklinn 7y ago
ast.parse(inspect.getsource(func)) doesn't work in the python shell though. You could also decompile the function's code object.

masklinn 7y ago

TBF I don't know how common it is as Python's AST API is pretty gnarly (not to mention really hard to work with / debug as it will usually segfault if you look at it funny). And IIRC the error messages coming from there are significantly worse than those coming from source. Third-party packages do make that somewhat better, but still not great. IIRC Jinja compiles templates to Python source code then compiles that to bytecode using the relevant builtins because it's simpler & more reliable than doing straight AST manipulations.

pritambaral 7y ago

> get the AST (a data structure AFAIK), manipulate it in a sane fashion in Python, compile it, return the result. No string munching eval require I use both Lisps and Python, and especially after learning to use Lisps I've wanted to do this in Python, so I looked for the ability to do what you describe here. While it is technically possible to do this (like most things) in Python, it is quite the effort and nowhere near ergonomic. Among all the cases where I could really use this, I found it worth the effort to instead: 1. Manipulate & generate code statically (from ASTs), as a build-time / dev-time step, using redbaron etc.; or 2. Just use a Lisp (specifically, Hy). Both methods involved writing quite a lot of code. In the former: to load, parse, match, and transform ASTs; in the latter: to port existing code (the code I originally wanted to manipulate) from Python to Hy. ---- The reason for all this is plainly the fact that Lisps are designed for this, while Python is not. Before learning Lisp, I used to wonder what could be so different in the language itself that could possibly make it that much easier. Now I use a Lisp-in-Python in production precisely because of this.

mark_l_watson 7y ago
Thank you, interesting comments, especially switching over to the Hy language for production. I got very excited about Hy, then an update took away a built in “let” and that turned me off a bit. Are there any good directions for patching in the old code supporting “let”?
marmaduke 7y ago
This is true, but I wanted to correct factual errors in the article. If you or your team are Lisp users then Hy or Clojure would be good choices, but if you are just trying to create a transformation for existing Python codebases, the decorator approach is a good shoehorn. On the other hand, you can also interpret the difficulty of AST transforms as a signal from the language designers (had van Rossum really never heard of Lisp?) that they hurt readability and their use should require a little ceremony.
- pritambaral 7y ago
  I'm sorry, but I don't agree at all. > if you are just trying to create a transformation for existing Python codebases, the decorator approach is a good shoehorn. For transformations that need to modify the code, decorators are insufficient. Decorators have their value, and I have built good interfaces using decorators, which would have been rather difficult without them, but all of that was because decorators help _wrap_, not _transform_ code. When I had a need to transform code, I looked for the right tool, and the takeaways of my searches are mentioned in my original comment. > you can also interpret the difficulty of AST transforms as a signal from the language designers (had van Rossum really never heard of Lisp?) that they hurt readability and their use should require a little ceremony. Nobody's claiming Guido had never heard of Lisp, or any non-lispy-language designer for that matter. Language designers make languages for varied reasons. Guido's reasons actually happen to be documented, and the only reason Python isn't homoiconic is because Guido didn't have a need for it, not because he explicitly wanted AST transforms to "require a little ceremony". In fact, Python AST transforms do not merely "require a little ceremony". They are actually quite cumbersome. The standard library actually provides no help at all in transforms, making it necessary for libraries like Rope, RedBaron, and Astor to be written.

ertucetin 7y ago

Python is not as flexible as Lisp in a manner of AST manipulation

smitty1e 7y ago
Python's approach seems to be: expose functionality like the AST for the 31337 ninjas that really want to go there, while maintaining intense focus on producing a KISS tool for the other 95% of its audience.
- jacobobryant 7y ago
  A lot of (most?) of the benefit of macros comes not from writing them yourself, but from using macros that others have written. Even if only 5% of the people are writing macros, that 5% likely includes many library authors. Making it harder for them impacts their downstream users too, even if those people aren't actually writing macros. IMO the macros in Clojure make it easier for library authors to produce KISS, high-level interfaces for the rest of us.
- TeMPOraL 7y ago
  This ultimately impacts code quality negatively. Macros aren't a tool "for the 31337 ninjas", not more than class inheritance or generic types, or hell, even procedures. Macros solve a particular class of problems - they eliminate bloat and boilerplate, allowing you to express your code closer to the way you think about it. Much like classes can be used as data transfer objects or to build out a large inheritance tree, macros can be used tactically - to DRY up things you otherwise couldn't, or introduce some readability improvement - or strategically, to lift the code up another layer of abstraction, or create DSLs that let you write your code much closer to the problem domain. Because Lisp supports them in an ergonomic way, macros are not seen as something special, but just another tool in the toolbox.

lenkite 7y ago

You might enjoy this talk- Live Coding a Compiler using Clojure at https://youtu.be/_lMltUNB6Yw?list=PLhYmIiHOMWoEgJEvgkmUe8D0agxy_T2vR https://youtu.be/_lMltUNB6Yw?list=PLhYmIiHOMWoEgJEvgkmUe8D0a...

kopos 7y ago

A python decorator is just a syntactic sugar for : f = decorator(f) Afaik, the decorator deals with the internal function as a black box. But with lisp being a lisp, the macro writer can choose the level of blackboxiness they want to deal with. Note: I love both Python and Clojure and use them extensively.

dangerbird2 7y ago
As others have hinted at, higher-order functions like decorators can inspect and manipulate the internals of a "black box" function object with the ast module[1]. This would be comparable to a lisp function taking a quoted form, expanding and evaling it at runtime as an ad-hoc macro funcitonality as is common in more bare-bones lisp implementations. Aside from the ease of manipulating quoted forms (i.e. asts) in Lisp and Clojure, a big part of the power of defmacro and kin is that Lisps distinguish (depending on implementation) between read-time, macroexpand-time, compile time, and eval time, giving the macro author control over when and where the macro is executed as well as how. As an example, this makes things like lazy evaluation (without requiring users to explicitly "thunk" values into functions) trivial to implement in lisp [1] https://docs.python.org/3/library/ast.html https://docs.python.org/3/library/ast.html
- kopos 7y ago
  Thank you, that was very informative. I do remember that Python had the ast module for a rules engine implementation, but I felt it was too cumbersome.. Thank you.

duckerude 7y ago

While you can manipulate Python's AST, you can't do it in a way that looks like adding new syntax. In Clojure you can write a list comprehension as `(for [x (range 10)] x)`. In Python you can write it as `[x for x in range(10)]`. Imagine these didn't exist, and you wanted to implement them yourself. In Clojure this is possible. Its list comprehensions are implemented as a Clojure macro: https://github.com/clojure/clojure/blob/b98ba84/src/clj/clojure/core.clj#L4590 https://github.com/clojure/clojure/blob/b98ba84/src/clj/cloj... In Python there are several problems. The first is that you can't easily pull the decorator trick on arbitrary expressions. Not even with a lambda function, at least without doing a lot of work - inspect.getsource doesn't handle those properly. But let's lower the bar, and replicate `l = [x for x in range(10)]` instead. You can do that with a decorator and an ordinary function. Then the problem becomes that you can only use existing syntax in the function body. Syntax that already means something else. If you put an ordinary list comprehension in the function body then you get a syntax error when the file is parsed, long before the decorator has a chance to do anything. So you have to make something up yourself. You might end up with something like this: @list_comp def l(): x: range(10) x Or, of course, this: l = list(map(lambda x: x, range(10))) Neither of these are as convenient or readable as real list comprehensions. But if you want those, you're at the whims of the language. You can't just add them yourself. I don't know how big of a deal this actually is. I've never used Clojure, and I've barely used other Lisps. Expressing what I want in Python is easy enough in practice (but maybe I just don't know any better?). But there's certainly something significant there that Python can't do.

zcid 7y ago
I heard somebody make a similar argument one time in regards to Common Lisp. He said that with Java you would have to wait several years sometimes to get a feature (assuming the language developers actually cared about the feature), but with Lisp, you could just implement the feature yourself. I've always felt this is one of the strongest arguments for using a lisp-style language. Edit: this is possible too in languages like Elixer and Ruby using macros, but eventually the readability suffers and debugging complex macros in those languages can be a nightmare.
- deleted 7y ago

ken 7y ago

Python has become one of the most popular languages. I think they're just using it as an example of "here's a language you probably know which doesn't support this" -- or, as you note, kind of supports it now but makes it a real pain because it's not native to the syntax.

amelius 7y ago

Python's biggest problem wrt functional programming: multiline inline lambdas are not possible.

didibus 7y ago

It's more that code and data have the same representation. For example, where is the Json of Python? Where is the Python object notation which is both how you represent Python Objects in code as well as when serialized to data over a wire? [1 2 3] is both the serialized form of a vector as well as valid code to create a vector of itself. Now in JS, only object data is homoiconic. In Lisps, even code expressions are homoiconic, thus (+ 1 2) is the serialized form of the list of function + and args 1 and 2, as well as valid code to add 1 and 2 together. You do not need macros for homoiconicity. Macros is not what makes a language homoiconic. But once your language is homoiconic, it is very easy to add macros to it, and macros become much simpler to write. Which is why Lisps have macros as well.

iLemming 7y ago

> I'm all for Lisp but this just isn't convincing Nothing will convince you until you give it a heartfelt try. The joy from using Clojure is a real thing. Unfortunately, there are not too many languages out there you can say the same thing about (including Python). Clojure is not absolutely frustration-free PL, but it maintains a good balance between "fun" and "profit".