5 ms·

When I first started writing asm code, it was accepted to write self-modifying code (programmatically create new code, then jmp to it.) Then I discovered and wa

When I first started writing asm code, it was accepted to write self-modifying code (programmatically create new code, then jmp to it.) Then I discovered and was told it was a bad idea. So that's how I view Lisp macros - creating code on the fly and executing it. When I was studying this issue, I came across femtolisp, which is a standalone Lisp by the author of Julia, Jeff Bezanson. Julia has metaprogramming so I wanted to look at it. In this video [1] Professor Steve G. Johnson explains metaprogramming (around 10 min mark) and says "most of the time, don't do metaprogramming." He points out in the base Julia there are 36,798 methods, but only 138 macros and 14 generated functions (.04% of the code), so I have the impression it is still frowned upon. [1] https://www.youtube.com/watch?v=mSgXWpvQEHE https://www.youtube.com/watch?v=mSgXWpvQEHE

greggyb 5y ago
> So that's how I view Lisp macros - creating code on the fly and executing it. This is an incorrect assumption. In compiled Lisps[0], macros are a compile-time construct, which manipulate the data structures that represent your code[1]. All manipulation occurs at compile-time. In interpreted Lisps, macro evaluation is temporally intertwined with program execution, but the two phases are logically distinct. [0]: Compilation time in a Lisp is not necessarily similar to compilation in other languages. Working at a REPL does not imply that interpretation is happening. Clojure is an example of a Lisp that is exclusively compiled. Even when sending individual expressions to the REPL, they are compiled before being evaluated. [1]: From a reply elsewhere in this thread: https://youtu.be/P76Vbsk_3J0?t=1333 https://youtu.be/P76Vbsk_3J0?t=1333 See this video for an overview of the core of Clojure. The link starts with an overview of Clojure's data types and data structures (skipping the intro for the reasons behind Clojure). This leads directly into the evaluation model, which is based on the data structures of Clojure. Watch through ~1h:30m to get a feel for what macros are used for in Clojure's core. This gives a very good overview of the evaluation model for macros; this is generally applicable to Lisp macros outside of Clojure as well, though there are nuances to different dialects. I recommend that you start at the linked time, but if you want to jump straight to the evaluation model, you can jump to ~47m.
- mwattsun 5y ago
  Thank you for taking the time to answer! At compile time vs at runtime is an important distinction that I need to think about. I'll start the video from the beginning because I'm curious about Clojure and most things Rich Hickey does, since I liked is Simple Made Easy video so much.
- mwattsun 5y ago
  So Clojure programs are data structures, and you can pass a structure to a macro to rewrite it into something the compiler can evaluate. In this way you can extend the language. You can build your own DSL that solves your particular problem in an organic way that grows as you go along. If I understand that correctly I don't get it, because I would still have to write the macro, which in C# I would write as a function that I call in my source code, which doesn't grow the syntax of C#, but grows the program to a DSL that solves my problem, albeit not as nicely as in a REPL. Maybe this is all about the REPL and I don't use them or see any appeal in them. I like to write my text, look at it, and hit run. You write a REPL line and it disappears. I'll keep learning. I'm sure the light bulb will go off eventually.
  - greggyb 5y ago
    A canonical example of macro capabilities that methods/functions cannot recreate is a short-circuiting conditional. Clojure's only built-in conditional operator is 'if'. Despite this, we have nice short-circuiting or, and, when, unless, and other conditional operations in Clojure, defined as macros. Clojure (and C# and most languages) eagerly evaluate their function arguments. You cannot write a function that short circuits. Let's consider the following Clojure: (when false (infinite-loop)) If when is a function, it's compiled to the appropriate instructions to evaluate both arguments at call-time. This will cause our program to hang on (infinite-loop). But, when is a macro, which means that during compilation time, the compiler defers to the when macro. The when macro is near-trivial. (defmacro when "Evaluates test. If logical true, evaluates body in an implicit do." {:added "1.0"} [test & body] (list 'if test (cons 'do body))) So the compiler has encountered when and sends the data structure of the form to this macro to evaluate. The result of this macro evaluation is passed back to the compiler. So, when receives a list, '(false (infinite-loop)). when returns a list to the compiler, '(if false (do (infinite-loop)). The compiler sees that there is no more macro expansion to be done, so it emits the appropriate code to execute that operation. When we get to run-time, the if happily short-circuits and this ends up being a no-op. This sort of conditional macro tends to be trivial in implementation, but highlights the difference between an eagerly evaluated argument to a function and a syntactic form passed to a macro for transformation at compile time. And if Clojure lacked a conditional construct, you could happily add it. Continuing with the comparison between Clojure and C#. C# gained async as a new keyword which required compiler modifications to implement. Clojure got core.async as a library. Anyone could have implemented core.async on their own. Now, Clojure's core.async is CSP-style (similar to Go) and C#'s rewrites your code into a state machine on your behalf, which basically pretties up callback handlers for you. If you prefer that C# async style to CSP, you can introduce that construct in Clojure yourself with something that looks as "native" as core.async. If you want to use CSP in C#, you cannot create any syntax for this and so will never be able to make something feel native as async does. Or another example would be something like C#'s using construct for IDisposables. If you wanted to implement using in a C# without it, you couldn't. You cannot create new control flow constructs. Something similar with Java AutoClosables is implemented as a macro in Clojure as well (again, if it didn't exist in the language, you can add it just like below). You can implement arbitrary control flow in Lisp macros in a way that would require syntax and compiler modifications in other languages. (defmacro with-open "bindings => [name init ...] Evaluates body in a try expression with names bound to the values of the inits, and a finally clause that calls (.close name) on each name in reverse order." {:added "1.0"} [bindings & body] (assert-args (vector? bindings) "a vector for its binding" (even? (count bindings)) "an even number of forms in binding vector") (cond (= (count bindings) 0) `(do ~@body) (symbol? (bindings 0)) `(let ~(subvec bindings 0 2) (try (with-open ~(subvec bindings 2) ~@body) (finally (. ~(bindings 0) close)))) :else (throw (IllegalArgumentException. "with-open only allows Symbols in bindings")))) I hope these examples help to illustrate the differences. If you're looking for more examples, you can take a look at the Clojure source to see what functionality is implemented by macros. https://github.com/clojure/clojure/search?q=defmacro&type=code https://github.com/clojure/clojure/search?q=defmacro&type=co...
Zak 5y ago
Writing macros in Lisp when you don't have to is also generally discouraged. They're typically harder to write and debug than functions, so their use is best reserved for situations when the opposite would be true.
- dreamcompiler 5y ago
  Exactly this. New Lispers sometimes overuse macros because they're so powerful, but macro overuse can make your code impossible to debug or maintain. At the same time, when you really need a macro, you really, really need a macro because it can do things a function cannot like controlling when or if its arguments are evaluated. There are multiple reasons to use a macro rather than a function. I won't describe them here* but suffice it to say that the statement "macros are no longer useful in modern programming" is completely false. *See pg's _On Lisp_ for an in-depth discussion of macros.
adgjlsfhk1 5y ago
If you want to see some very productive uses of metaprogramming in Julia, check out Jump.jl or DataframesMeta.jl. Both use metaprogramming to create their own little DSLs that make expressing problems in a specific form very easy.
ehaliewicz2 5y ago
Self-modifying code isn't a bad idea. Like everything else, it has it's place, just not typically on modern computers due to breaking optimizations. What you describe is actually used in a lot of modern software, usually self-modifying code refers to code that modifies itself while running, typically in a loop, not just generating fresh code :)
- adgjlsfhk1 5y ago
  In my opinion, compile time codegen is way better than runtime codegen. Unless you are exploring genetic algorithms, runtime codegen is slow, brittle, and undebugable. Compile time codegen, however, can produce ridiculously fast code (see FFTW), and is much easier to debug because you can look at the generated code.
  - mwattsun 5y ago
    > brittle and undebuggable That's why I stopped doing it, but I totally get that in low level programming that it comes in very handy in some circumstances.
sundarurfriend 5y ago
> Professor Steve G. Johnson explains metaprogramming (around 10 min mark) and says "most of the time, don't do metaprogramming." As someone who has given similar advice when teaching metaprogramming in the past, I'll add some context to that: most people new to metaprogramming seem to reach for it all the time for some reason, in situations where the usual programming constructs are perfectly adequate. Imagine if any time you introduce people to multithreading, they start writing all their code as parallel threaded code (which happens to some degree, but is limited by factors include what a PITA it is). That's the context in which such warnings are deemed necessary. Just like multithreading, manual memory management, and other great-power-great-responsibility programming concepts, macros shouldn't be the first tool you reach for, but when you need them you really need them, and they can make your life so much easier.

greggyb 5y ago

> So that's how I view Lisp macros - creating code on the fly and executing it. This is an incorrect assumption. In compiled Lisps[0], macros are a compile-time construct, which manipulate the data structures that represent your code[1]. All manipulation occurs at compile-time. In interpreted Lisps, macro evaluation is temporally intertwined with program execution, but the two phases are logically distinct. [0]: Compilation time in a Lisp is not necessarily similar to compilation in other languages. Working at a REPL does not imply that interpretation is happening. Clojure is an example of a Lisp that is exclusively compiled. Even when sending individual expressions to the REPL, they are compiled before being evaluated. [1]: From a reply elsewhere in this thread: https://youtu.be/P76Vbsk_3J0?t=1333 https://youtu.be/P76Vbsk_3J0?t=1333 See this video for an overview of the core of Clojure. The link starts with an overview of Clojure's data types and data structures (skipping the intro for the reasons behind Clojure). This leads directly into the evaluation model, which is based on the data structures of Clojure. Watch through ~1h:30m to get a feel for what macros are used for in Clojure's core. This gives a very good overview of the evaluation model for macros; this is generally applicable to Lisp macros outside of Clojure as well, though there are nuances to different dialects. I recommend that you start at the linked time, but if you want to jump straight to the evaluation model, you can jump to ~47m.

mwattsun 5y ago
Thank you for taking the time to answer! At compile time vs at runtime is an important distinction that I need to think about. I'll start the video from the beginning because I'm curious about Clojure and most things Rich Hickey does, since I liked is Simple Made Easy video so much.
mwattsun 5y ago
So Clojure programs are data structures, and you can pass a structure to a macro to rewrite it into something the compiler can evaluate. In this way you can extend the language. You can build your own DSL that solves your particular problem in an organic way that grows as you go along. If I understand that correctly I don't get it, because I would still have to write the macro, which in C# I would write as a function that I call in my source code, which doesn't grow the syntax of C#, but grows the program to a DSL that solves my problem, albeit not as nicely as in a REPL. Maybe this is all about the REPL and I don't use them or see any appeal in them. I like to write my text, look at it, and hit run. You write a REPL line and it disappears. I'll keep learning. I'm sure the light bulb will go off eventually.
- greggyb 5y ago
  A canonical example of macro capabilities that methods/functions cannot recreate is a short-circuiting conditional. Clojure's only built-in conditional operator is 'if'. Despite this, we have nice short-circuiting or, and, when, unless, and other conditional operations in Clojure, defined as macros. Clojure (and C# and most languages) eagerly evaluate their function arguments. You cannot write a function that short circuits. Let's consider the following Clojure: (when false (infinite-loop)) If when is a function, it's compiled to the appropriate instructions to evaluate both arguments at call-time. This will cause our program to hang on (infinite-loop). But, when is a macro, which means that during compilation time, the compiler defers to the when macro. The when macro is near-trivial. (defmacro when "Evaluates test. If logical true, evaluates body in an implicit do." {:added "1.0"} [test & body] (list 'if test (cons 'do body))) So the compiler has encountered when and sends the data structure of the form to this macro to evaluate. The result of this macro evaluation is passed back to the compiler. So, when receives a list, '(false (infinite-loop)). when returns a list to the compiler, '(if false (do (infinite-loop)). The compiler sees that there is no more macro expansion to be done, so it emits the appropriate code to execute that operation. When we get to run-time, the if happily short-circuits and this ends up being a no-op. This sort of conditional macro tends to be trivial in implementation, but highlights the difference between an eagerly evaluated argument to a function and a syntactic form passed to a macro for transformation at compile time. And if Clojure lacked a conditional construct, you could happily add it. Continuing with the comparison between Clojure and C#. C# gained async as a new keyword which required compiler modifications to implement. Clojure got core.async as a library. Anyone could have implemented core.async on their own. Now, Clojure's core.async is CSP-style (similar to Go) and C#'s rewrites your code into a state machine on your behalf, which basically pretties up callback handlers for you. If you prefer that C# async style to CSP, you can introduce that construct in Clojure yourself with something that looks as "native" as core.async. If you want to use CSP in C#, you cannot create any syntax for this and so will never be able to make something feel native as async does. Or another example would be something like C#'s using construct for IDisposables. If you wanted to implement using in a C# without it, you couldn't. You cannot create new control flow constructs. Something similar with Java AutoClosables is implemented as a macro in Clojure as well (again, if it didn't exist in the language, you can add it just like below). You can implement arbitrary control flow in Lisp macros in a way that would require syntax and compiler modifications in other languages. (defmacro with-open "bindings => [name init ...] Evaluates body in a try expression with names bound to the values of the inits, and a finally clause that calls (.close name) on each name in reverse order." {:added "1.0"} [bindings & body] (assert-args (vector? bindings) "a vector for its binding" (even? (count bindings)) "an even number of forms in binding vector") (cond (= (count bindings) 0) `(do ~@body) (symbol? (bindings 0)) `(let ~(subvec bindings 0 2) (try (with-open ~(subvec bindings 2) ~@body) (finally (. ~(bindings 0) close)))) :else (throw (IllegalArgumentException. "with-open only allows Symbols in bindings")))) I hope these examples help to illustrate the differences. If you're looking for more examples, you can take a look at the Clojure source to see what functionality is implemented by macros. https://github.com/clojure/clojure/search?q=defmacro&type=code https://github.com/clojure/clojure/search?q=defmacro&type=co...

Zak 5y ago

Writing macros in Lisp when you don't have to is also generally discouraged. They're typically harder to write and debug than functions, so their use is best reserved for situations when the opposite would be true.

dreamcompiler 5y ago
Exactly this. New Lispers sometimes overuse macros because they're so powerful, but macro overuse can make your code impossible to debug or maintain. At the same time, when you really need a macro, you really, really need a macro because it can do things a function cannot like controlling when or if its arguments are evaluated. There are multiple reasons to use a macro rather than a function. I won't describe them here* but suffice it to say that the statement "macros are no longer useful in modern programming" is completely false. *See pg's _On Lisp_ for an in-depth discussion of macros.

adgjlsfhk1 5y ago

If you want to see some very productive uses of metaprogramming in Julia, check out Jump.jl or DataframesMeta.jl. Both use metaprogramming to create their own little DSLs that make expressing problems in a specific form very easy.

ehaliewicz2 5y ago

Self-modifying code isn't a bad idea. Like everything else, it has it's place, just not typically on modern computers due to breaking optimizations. What you describe is actually used in a lot of modern software, usually self-modifying code refers to code that modifies itself while running, typically in a loop, not just generating fresh code :)

adgjlsfhk1 5y ago
In my opinion, compile time codegen is way better than runtime codegen. Unless you are exploring genetic algorithms, runtime codegen is slow, brittle, and undebugable. Compile time codegen, however, can produce ridiculously fast code (see FFTW), and is much easier to debug because you can look at the generated code.
- mwattsun 5y ago
  > brittle and undebuggable That's why I stopped doing it, but I totally get that in low level programming that it comes in very handy in some circumstances.

sundarurfriend 5y ago

> Professor Steve G. Johnson explains metaprogramming (around 10 min mark) and says "most of the time, don't do metaprogramming." As someone who has given similar advice when teaching metaprogramming in the past, I'll add some context to that: most people new to metaprogramming seem to reach for it all the time for some reason, in situations where the usual programming constructs are perfectly adequate. Imagine if any time you introduce people to multithreading, they start writing all their code as parallel threaded code (which happens to some degree, but is limited by factors include what a PITA it is). That's the context in which such warnings are deemed necessary. Just like multithreading, manual memory management, and other great-power-great-responsibility programming concepts, macros shouldn't be the first tool you reach for, but when you need them you really need them, and they can make your life so much easier.