3 ms·

> So that's how I view Lisp macros - creating code on the fly and executing it. This is an incorrect assumption. In compiled Lisps[0], macros are a compile-tim

> So that's how I view Lisp macros - creating code on the fly and executing it. This is an incorrect assumption. In compiled Lisps[0], macros are a compile-time construct, which manipulate the data structures that represent your code[1]. All manipulation occurs at compile-time. In interpreted Lisps, macro evaluation is temporally intertwined with program execution, but the two phases are logically distinct. [0]: Compilation time in a Lisp is not necessarily similar to compilation in other languages. Working at a REPL does not imply that interpretation is happening. Clojure is an example of a Lisp that is exclusively compiled. Even when sending individual expressions to the REPL, they are compiled before being evaluated. [1]: From a reply elsewhere in this thread: https://youtu.be/P76Vbsk_3J0?t=1333 https://youtu.be/P76Vbsk_3J0?t=1333 See this video for an overview of the core of Clojure. The link starts with an overview of Clojure's data types and data structures (skipping the intro for the reasons behind Clojure). This leads directly into the evaluation model, which is based on the data structures of Clojure. Watch through ~1h:30m to get a feel for what macros are used for in Clojure's core. This gives a very good overview of the evaluation model for macros; this is generally applicable to Lisp macros outside of Clojure as well, though there are nuances to different dialects. I recommend that you start at the linked time, but if you want to jump straight to the evaluation model, you can jump to ~47m.

mwattsun 5y ago
Thank you for taking the time to answer! At compile time vs at runtime is an important distinction that I need to think about. I'll start the video from the beginning because I'm curious about Clojure and most things Rich Hickey does, since I liked is Simple Made Easy video so much.
mwattsun 5y ago
So Clojure programs are data structures, and you can pass a structure to a macro to rewrite it into something the compiler can evaluate. In this way you can extend the language. You can build your own DSL that solves your particular problem in an organic way that grows as you go along. If I understand that correctly I don't get it, because I would still have to write the macro, which in C# I would write as a function that I call in my source code, which doesn't grow the syntax of C#, but grows the program to a DSL that solves my problem, albeit not as nicely as in a REPL. Maybe this is all about the REPL and I don't use them or see any appeal in them. I like to write my text, look at it, and hit run. You write a REPL line and it disappears. I'll keep learning. I'm sure the light bulb will go off eventually.
- greggyb 5y ago
  A canonical example of macro capabilities that methods/functions cannot recreate is a short-circuiting conditional. Clojure's only built-in conditional operator is 'if'. Despite this, we have nice short-circuiting or, and, when, unless, and other conditional operations in Clojure, defined as macros. Clojure (and C# and most languages) eagerly evaluate their function arguments. You cannot write a function that short circuits. Let's consider the following Clojure: (when false (infinite-loop)) If when is a function, it's compiled to the appropriate instructions to evaluate both arguments at call-time. This will cause our program to hang on (infinite-loop). But, when is a macro, which means that during compilation time, the compiler defers to the when macro. The when macro is near-trivial. (defmacro when "Evaluates test. If logical true, evaluates body in an implicit do." {:added "1.0"} [test & body] (list 'if test (cons 'do body))) So the compiler has encountered when and sends the data structure of the form to this macro to evaluate. The result of this macro evaluation is passed back to the compiler. So, when receives a list, '(false (infinite-loop)). when returns a list to the compiler, '(if false (do (infinite-loop)). The compiler sees that there is no more macro expansion to be done, so it emits the appropriate code to execute that operation. When we get to run-time, the if happily short-circuits and this ends up being a no-op. This sort of conditional macro tends to be trivial in implementation, but highlights the difference between an eagerly evaluated argument to a function and a syntactic form passed to a macro for transformation at compile time. And if Clojure lacked a conditional construct, you could happily add it. Continuing with the comparison between Clojure and C#. C# gained async as a new keyword which required compiler modifications to implement. Clojure got core.async as a library. Anyone could have implemented core.async on their own. Now, Clojure's core.async is CSP-style (similar to Go) and C#'s rewrites your code into a state machine on your behalf, which basically pretties up callback handlers for you. If you prefer that C# async style to CSP, you can introduce that construct in Clojure yourself with something that looks as "native" as core.async. If you want to use CSP in C#, you cannot create any syntax for this and so will never be able to make something feel native as async does. Or another example would be something like C#'s using construct for IDisposables. If you wanted to implement using in a C# without it, you couldn't. You cannot create new control flow constructs. Something similar with Java AutoClosables is implemented as a macro in Clojure as well (again, if it didn't exist in the language, you can add it just like below). You can implement arbitrary control flow in Lisp macros in a way that would require syntax and compiler modifications in other languages. (defmacro with-open "bindings => [name init ...] Evaluates body in a try expression with names bound to the values of the inits, and a finally clause that calls (.close name) on each name in reverse order." {:added "1.0"} [bindings & body] (assert-args (vector? bindings) "a vector for its binding" (even? (count bindings)) "an even number of forms in binding vector") (cond (= (count bindings) 0) `(do ~@body) (symbol? (bindings 0)) `(let ~(subvec bindings 0 2) (try (with-open ~(subvec bindings 2) ~@body) (finally (. ~(bindings 0) close)))) :else (throw (IllegalArgumentException. "with-open only allows Symbols in bindings")))) I hope these examples help to illustrate the differences. If you're looking for more examples, you can take a look at the Clojure source to see what functionality is implemented by macros. https://github.com/clojure/clojure/search?q=defmacro&type=code https://github.com/clojure/clojure/search?q=defmacro&type=co...

mwattsun 5y ago

Thank you for taking the time to answer! At compile time vs at runtime is an important distinction that I need to think about. I'll start the video from the beginning because I'm curious about Clojure and most things Rich Hickey does, since I liked is Simple Made Easy video so much.

mwattsun 5y ago

So Clojure programs are data structures, and you can pass a structure to a macro to rewrite it into something the compiler can evaluate. In this way you can extend the language. You can build your own DSL that solves your particular problem in an organic way that grows as you go along. If I understand that correctly I don't get it, because I would still have to write the macro, which in C# I would write as a function that I call in my source code, which doesn't grow the syntax of C#, but grows the program to a DSL that solves my problem, albeit not as nicely as in a REPL. Maybe this is all about the REPL and I don't use them or see any appeal in them. I like to write my text, look at it, and hit run. You write a REPL line and it disappears. I'll keep learning. I'm sure the light bulb will go off eventually.

greggyb 5y ago
A canonical example of macro capabilities that methods/functions cannot recreate is a short-circuiting conditional. Clojure's only built-in conditional operator is 'if'. Despite this, we have nice short-circuiting or, and, when, unless, and other conditional operations in Clojure, defined as macros. Clojure (and C# and most languages) eagerly evaluate their function arguments. You cannot write a function that short circuits. Let's consider the following Clojure: (when false (infinite-loop)) If when is a function, it's compiled to the appropriate instructions to evaluate both arguments at call-time. This will cause our program to hang on (infinite-loop). But, when is a macro, which means that during compilation time, the compiler defers to the when macro. The when macro is near-trivial. (defmacro when "Evaluates test. If logical true, evaluates body in an implicit do." {:added "1.0"} [test & body] (list 'if test (cons 'do body))) So the compiler has encountered when and sends the data structure of the form to this macro to evaluate. The result of this macro evaluation is passed back to the compiler. So, when receives a list, '(false (infinite-loop)). when returns a list to the compiler, '(if false (do (infinite-loop)). The compiler sees that there is no more macro expansion to be done, so it emits the appropriate code to execute that operation. When we get to run-time, the if happily short-circuits and this ends up being a no-op. This sort of conditional macro tends to be trivial in implementation, but highlights the difference between an eagerly evaluated argument to a function and a syntactic form passed to a macro for transformation at compile time. And if Clojure lacked a conditional construct, you could happily add it. Continuing with the comparison between Clojure and C#. C# gained async as a new keyword which required compiler modifications to implement. Clojure got core.async as a library. Anyone could have implemented core.async on their own. Now, Clojure's core.async is CSP-style (similar to Go) and C#'s rewrites your code into a state machine on your behalf, which basically pretties up callback handlers for you. If you prefer that C# async style to CSP, you can introduce that construct in Clojure yourself with something that looks as "native" as core.async. If you want to use CSP in C#, you cannot create any syntax for this and so will never be able to make something feel native as async does. Or another example would be something like C#'s using construct for IDisposables. If you wanted to implement using in a C# without it, you couldn't. You cannot create new control flow constructs. Something similar with Java AutoClosables is implemented as a macro in Clojure as well (again, if it didn't exist in the language, you can add it just like below). You can implement arbitrary control flow in Lisp macros in a way that would require syntax and compiler modifications in other languages. (defmacro with-open "bindings => [name init ...] Evaluates body in a try expression with names bound to the values of the inits, and a finally clause that calls (.close name) on each name in reverse order." {:added "1.0"} [bindings & body] (assert-args (vector? bindings) "a vector for its binding" (even? (count bindings)) "an even number of forms in binding vector") (cond (= (count bindings) 0) `(do ~@body) (symbol? (bindings 0)) `(let ~(subvec bindings 0 2) (try (with-open ~(subvec bindings 2) ~@body) (finally (. ~(bindings 0) close)))) :else (throw (IllegalArgumentException. "with-open only allows Symbols in bindings")))) I hope these examples help to illustrate the differences. If you're looking for more examples, you can take a look at the Clojure source to see what functionality is implemented by macros. https://github.com/clojure/clojure/search?q=defmacro&type=code https://github.com/clojure/clojure/search?q=defmacro&type=co...