3 ms·

So Clojure programs are data structures, and you can pass a structure to a macro to rewrite it into something the compiler can evaluate. In this way you can ext

So Clojure programs are data structures, and you can pass a structure to a macro to rewrite it into something the compiler can evaluate. In this way you can extend the language. You can build your own DSL that solves your particular problem in an organic way that grows as you go along. If I understand that correctly I don't get it, because I would still have to write the macro, which in C# I would write as a function that I call in my source code, which doesn't grow the syntax of C#, but grows the program to a DSL that solves my problem, albeit not as nicely as in a REPL. Maybe this is all about the REPL and I don't use them or see any appeal in them. I like to write my text, look at it, and hit run. You write a REPL line and it disappears. I'll keep learning. I'm sure the light bulb will go off eventually.

greggyb 5y ago
A canonical example of macro capabilities that methods/functions cannot recreate is a short-circuiting conditional. Clojure's only built-in conditional operator is 'if'. Despite this, we have nice short-circuiting or, and, when, unless, and other conditional operations in Clojure, defined as macros. Clojure (and C# and most languages) eagerly evaluate their function arguments. You cannot write a function that short circuits. Let's consider the following Clojure: (when false (infinite-loop)) If when is a function, it's compiled to the appropriate instructions to evaluate both arguments at call-time. This will cause our program to hang on (infinite-loop). But, when is a macro, which means that during compilation time, the compiler defers to the when macro. The when macro is near-trivial. (defmacro when "Evaluates test. If logical true, evaluates body in an implicit do." {:added "1.0"} [test & body] (list 'if test (cons 'do body))) So the compiler has encountered when and sends the data structure of the form to this macro to evaluate. The result of this macro evaluation is passed back to the compiler. So, when receives a list, '(false (infinite-loop)). when returns a list to the compiler, '(if false (do (infinite-loop)). The compiler sees that there is no more macro expansion to be done, so it emits the appropriate code to execute that operation. When we get to run-time, the if happily short-circuits and this ends up being a no-op. This sort of conditional macro tends to be trivial in implementation, but highlights the difference between an eagerly evaluated argument to a function and a syntactic form passed to a macro for transformation at compile time. And if Clojure lacked a conditional construct, you could happily add it. Continuing with the comparison between Clojure and C#. C# gained async as a new keyword which required compiler modifications to implement. Clojure got core.async as a library. Anyone could have implemented core.async on their own. Now, Clojure's core.async is CSP-style (similar to Go) and C#'s rewrites your code into a state machine on your behalf, which basically pretties up callback handlers for you. If you prefer that C# async style to CSP, you can introduce that construct in Clojure yourself with something that looks as "native" as core.async. If you want to use CSP in C#, you cannot create any syntax for this and so will never be able to make something feel native as async does. Or another example would be something like C#'s using construct for IDisposables. If you wanted to implement using in a C# without it, you couldn't. You cannot create new control flow constructs. Something similar with Java AutoClosables is implemented as a macro in Clojure as well (again, if it didn't exist in the language, you can add it just like below). You can implement arbitrary control flow in Lisp macros in a way that would require syntax and compiler modifications in other languages. (defmacro with-open "bindings => [name init ...] Evaluates body in a try expression with names bound to the values of the inits, and a finally clause that calls (.close name) on each name in reverse order." {:added "1.0"} [bindings & body] (assert-args (vector? bindings) "a vector for its binding" (even? (count bindings)) "an even number of forms in binding vector") (cond (= (count bindings) 0) `(do ~@body) (symbol? (bindings 0)) `(let ~(subvec bindings 0 2) (try (with-open ~(subvec bindings 2) ~@body) (finally (. ~(bindings 0) close)))) :else (throw (IllegalArgumentException. "with-open only allows Symbols in bindings")))) I hope these examples help to illustrate the differences. If you're looking for more examples, you can take a look at the Clojure source to see what functionality is implemented by macros. https://github.com/clojure/clojure/search?q=defmacro&type=code https://github.com/clojure/clojure/search?q=defmacro&type=co...
- mwattsun 5y ago
  Thank you for the examples. I'm really curious to get to the bottom of this. My inspiration has been Erik Meijer, specifically his article "Confessions of a Used Programming Language Salesman. Getting the Masses Hooked on Haskell" [1]. If I understand you correctly, you are implementing lazy evaluation in your when macro, something you call short-circuiting. By dropping laziness in strict functional languages such as Scheme, SML, OCaml, Scala, and F#, the purity and semantic beauty of true functional programming is lost. When programming in Haskell, laziness is one of those things that you rely on all the time without noticing it; it is something you only realize once you miss it. As for async/await, Erik led the effort to put it in C# and then Dart. I'm assuming he wanted to bring Haskell Continuation Monads [2] to the popular languages. I'm building a hobby website, so I got into Blazor because I like C# and was interested in doing it client side. I gave up on it, not because of C# or bad implementation (it's actually great) but because Microsoft has a noisy and bureaucratic way of configuring code that drives me nuts. I went back to Dart because it is amazingly productive on the client side, but now this video [3] shared in this thread makes me think I should give ClojureScript a try. [1] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.72.868&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.72.... [2] https://www.haskellforall.com/2012/12/the-continuation-monad.html https://www.haskellforall.com/2012/12/the-continuation-monad... [3] https://vimeo.com/230220635 https://vimeo.com/230220635
  - greggyb 5y ago
    I'm happy to chat more if it's helpful (contact details in profile). I'll be honest that I didn't get Lisp for a long time, and stilll wouldn't claim true expertise. I attempted SICP a few times based on the glowing reviews from folks I admire. One of my paths to tech was via Paul Graham's writings, so I had a bias for Lisp early on. Two parts here. First, and briefly a note on short circuiting and lazy evaluation. Second, and longer, a further discussion of Lisp macros. First. Short circuiting is a special case of lazy evaluation, but is the most common exposure to lazy evaluation among programmers in the C family of languages. It is the property of conditional and boolean operations that evaluate only the minimum necessary to return a result. (or true IGNORED) only needs to evaluate the first operand to or, true, because the result of the or is true regardless of the value of IGNORED. This is common to all mainstream strictly evaluated languages I am aware of. https://en.wikipedia.org/wiki/Short-circuit_evaluation https://en.wikipedia.org/wiki/Short-circuit_evaluation Macros allow you to implement lazy evaluation in an otherwise strict language, but this is not a comprehensive explanation of their utility. Second. I've watched pretty much all of Rich Hickey's presentations a half dozen times or more. Somewhere in the mix of more practice, another time through SICP, and watching Rich Hickey I started getting what homoiconicity means and the value prop of Lisp. It's simultaneously very prosaic and very deep. Macros are pretty simple: 1. Access to the AST of a chunk of code for arbitrary processing. 2. At compile-time. This is not exclusive to Lisps. I think pretty much any mature language offers a library or a built-in mechanism for getting at the AST of arbitrary code in that language. The "at compile-time" part is not necessarily built in to other languages. You absolutely can have a build system that has two (or more) passes, the first building a macro expansion framework and library of macros and the second processing the rest of your source code and rewriting the AST of the code in those files to do macro expansion. Where Lisps are different: 1. The AST of the language is represented directly in the core data structures of the language. An operation in Lisp[0] is simply a list whose first element is an operator (one of a special operator, a macro, or a function in most Lisps) and whose remaining elements are operands to that operator. This makes AST manipulation easier, because you use the same functions/abstractions/interfaces as in normal programming, because you're directly using familiar data structures. 2. Separation of reader and evaluator (interpreter or compiler or either/both). The reader transforms textual representations into in-memory representations of values and data structures. The evaluator only ever receives these in-memory representations. Thus, it's easier to do programmatic manipulation of an AST. You never have to do code-gen of emitting source code glyphs to feed into a compiler that expects text input. Once past the reader, everything is an in-memory representation, so we never have to get back to text. That said, it's trivial to get back to text if you want, because Lisps also include a printer which takes an in-memory representation and prints the glyphs that represent their readable (i.e., can be read by the reader) textual representation. 3. The evaluation model explicitly supports tagging some code as a macro to go through macro expansion before being evaluated "normally". 4. It is idiomatic to use macros to solve code re-use problems that cannot be readily handled by standard function calls.[1] These four Lisp traits are not exclusive to Lisp, necessarily. As I mentioned above, you could build a macro expansion and code generation library that is part of a build process for any compiled language. Eval, in interpreted languages, is as expressive as Lisp macros, but is incredibly poorly supported and unergonomic in comparison to Lisp macros. It's an ergonomic and feasibility thing more than a capability thing. That said, POSIX shell is turing complete, so you don't need anything like C# or Lisp or Haskell or anything else we've discussed to get your computation done. The reaction you might have to comparing shell scripting to Haskell is not dissimilar to the reaction an experienced Lisp programmer might have to comparing AST manipulation and code gen through a library to Lisp's macro facilities. [0]: Clojure, specifically abstracts the idea of callable, and there are more callable things than special operators, macros, and functions. [1]: Languages such as Haskell and OCaml continue to push the boundaries of what can be handled by "standard function calls" (which term I hesitate to use, given its imprecision). That said, given that macros allow the use of a turing-complete language to generate inputs to the evaluator, they allow you to implement any language construct that is computable without modifying the evaluation infrastructure.
greggyb 5y ago
As for REPL driven development, it's not about typing code at a REPL. This video provides a pretty good example of working at a REPL in Clojure. Note that with the exception of one doc reference at the beginning, all code is typed directly into a file buffer. The editor provides shortcuts to send various forms to the REPL for evaluation. https://vimeo.com/230220635 https://vimeo.com/230220635
p_l 5y ago
Consider that macros generally don't expand syntax - because there is very little syntax anyway, and what you have is a tree of data and function calls. A Macro just lets you write a function that will be executed by compiler to manipulate such tree, meaning that you can have equivalent of adding syntax like "with/using" from some languages by writing a function that takes as arguments the object and a block of code, then re-emits the same block of code but adding object.Dispose() call at the end. Or whatever else you need. But they are still normal functions - it's the time of execution that changes.
- mwattsun 5y ago
  I generally never used macros when I wrote C because the C macro language is lacking, there are a lot of pitfalls and saving function call overhead is usually not that important. I think I'll try to get a new understanding of Lisp macros by trying to come up with a language feature in C# or Dart that I really want but can't implement without a lot of non-intuitive difficulty, but could easily in a Lisp style macro that the compiler could use to build in a new feature at compile time.

greggyb 5y ago

A canonical example of macro capabilities that methods/functions cannot recreate is a short-circuiting conditional. Clojure's only built-in conditional operator is 'if'. Despite this, we have nice short-circuiting or, and, when, unless, and other conditional operations in Clojure, defined as macros. Clojure (and C# and most languages) eagerly evaluate their function arguments. You cannot write a function that short circuits. Let's consider the following Clojure: (when false (infinite-loop)) If when is a function, it's compiled to the appropriate instructions to evaluate both arguments at call-time. This will cause our program to hang on (infinite-loop). But, when is a macro, which means that during compilation time, the compiler defers to the when macro. The when macro is near-trivial. (defmacro when "Evaluates test. If logical true, evaluates body in an implicit do." {:added "1.0"} [test & body] (list 'if test (cons 'do body))) So the compiler has encountered when and sends the data structure of the form to this macro to evaluate. The result of this macro evaluation is passed back to the compiler. So, when receives a list, '(false (infinite-loop)). when returns a list to the compiler, '(if false (do (infinite-loop)). The compiler sees that there is no more macro expansion to be done, so it emits the appropriate code to execute that operation. When we get to run-time, the if happily short-circuits and this ends up being a no-op. This sort of conditional macro tends to be trivial in implementation, but highlights the difference between an eagerly evaluated argument to a function and a syntactic form passed to a macro for transformation at compile time. And if Clojure lacked a conditional construct, you could happily add it. Continuing with the comparison between Clojure and C#. C# gained async as a new keyword which required compiler modifications to implement. Clojure got core.async as a library. Anyone could have implemented core.async on their own. Now, Clojure's core.async is CSP-style (similar to Go) and C#'s rewrites your code into a state machine on your behalf, which basically pretties up callback handlers for you. If you prefer that C# async style to CSP, you can introduce that construct in Clojure yourself with something that looks as "native" as core.async. If you want to use CSP in C#, you cannot create any syntax for this and so will never be able to make something feel native as async does. Or another example would be something like C#'s using construct for IDisposables. If you wanted to implement using in a C# without it, you couldn't. You cannot create new control flow constructs. Something similar with Java AutoClosables is implemented as a macro in Clojure as well (again, if it didn't exist in the language, you can add it just like below). You can implement arbitrary control flow in Lisp macros in a way that would require syntax and compiler modifications in other languages. (defmacro with-open "bindings => [name init ...] Evaluates body in a try expression with names bound to the values of the inits, and a finally clause that calls (.close name) on each name in reverse order." {:added "1.0"} [bindings & body] (assert-args (vector? bindings) "a vector for its binding" (even? (count bindings)) "an even number of forms in binding vector") (cond (= (count bindings) 0) `(do ~@body) (symbol? (bindings 0)) `(let ~(subvec bindings 0 2) (try (with-open ~(subvec bindings 2) ~@body) (finally (. ~(bindings 0) close)))) :else (throw (IllegalArgumentException. "with-open only allows Symbols in bindings")))) I hope these examples help to illustrate the differences. If you're looking for more examples, you can take a look at the Clojure source to see what functionality is implemented by macros. https://github.com/clojure/clojure/search?q=defmacro&type=code https://github.com/clojure/clojure/search?q=defmacro&type=co...

mwattsun 5y ago
Thank you for the examples. I'm really curious to get to the bottom of this. My inspiration has been Erik Meijer, specifically his article "Confessions of a Used Programming Language Salesman. Getting the Masses Hooked on Haskell" [1]. If I understand you correctly, you are implementing lazy evaluation in your when macro, something you call short-circuiting. By dropping laziness in strict functional languages such as Scheme, SML, OCaml, Scala, and F#, the purity and semantic beauty of true functional programming is lost. When programming in Haskell, laziness is one of those things that you rely on all the time without noticing it; it is something you only realize once you miss it. As for async/await, Erik led the effort to put it in C# and then Dart. I'm assuming he wanted to bring Haskell Continuation Monads [2] to the popular languages. I'm building a hobby website, so I got into Blazor because I like C# and was interested in doing it client side. I gave up on it, not because of C# or bad implementation (it's actually great) but because Microsoft has a noisy and bureaucratic way of configuring code that drives me nuts. I went back to Dart because it is amazingly productive on the client side, but now this video [3] shared in this thread makes me think I should give ClojureScript a try. [1] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.72.868&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.72.... [2] https://www.haskellforall.com/2012/12/the-continuation-monad.html https://www.haskellforall.com/2012/12/the-continuation-monad... [3] https://vimeo.com/230220635 https://vimeo.com/230220635
- greggyb 5y ago
  I'm happy to chat more if it's helpful (contact details in profile). I'll be honest that I didn't get Lisp for a long time, and stilll wouldn't claim true expertise. I attempted SICP a few times based on the glowing reviews from folks I admire. One of my paths to tech was via Paul Graham's writings, so I had a bias for Lisp early on. Two parts here. First, and briefly a note on short circuiting and lazy evaluation. Second, and longer, a further discussion of Lisp macros. First. Short circuiting is a special case of lazy evaluation, but is the most common exposure to lazy evaluation among programmers in the C family of languages. It is the property of conditional and boolean operations that evaluate only the minimum necessary to return a result. (or true IGNORED) only needs to evaluate the first operand to or, true, because the result of the or is true regardless of the value of IGNORED. This is common to all mainstream strictly evaluated languages I am aware of. https://en.wikipedia.org/wiki/Short-circuit_evaluation https://en.wikipedia.org/wiki/Short-circuit_evaluation Macros allow you to implement lazy evaluation in an otherwise strict language, but this is not a comprehensive explanation of their utility. Second. I've watched pretty much all of Rich Hickey's presentations a half dozen times or more. Somewhere in the mix of more practice, another time through SICP, and watching Rich Hickey I started getting what homoiconicity means and the value prop of Lisp. It's simultaneously very prosaic and very deep. Macros are pretty simple: 1. Access to the AST of a chunk of code for arbitrary processing. 2. At compile-time. This is not exclusive to Lisps. I think pretty much any mature language offers a library or a built-in mechanism for getting at the AST of arbitrary code in that language. The "at compile-time" part is not necessarily built in to other languages. You absolutely can have a build system that has two (or more) passes, the first building a macro expansion framework and library of macros and the second processing the rest of your source code and rewriting the AST of the code in those files to do macro expansion. Where Lisps are different: 1. The AST of the language is represented directly in the core data structures of the language. An operation in Lisp[0] is simply a list whose first element is an operator (one of a special operator, a macro, or a function in most Lisps) and whose remaining elements are operands to that operator. This makes AST manipulation easier, because you use the same functions/abstractions/interfaces as in normal programming, because you're directly using familiar data structures. 2. Separation of reader and evaluator (interpreter or compiler or either/both). The reader transforms textual representations into in-memory representations of values and data structures. The evaluator only ever receives these in-memory representations. Thus, it's easier to do programmatic manipulation of an AST. You never have to do code-gen of emitting source code glyphs to feed into a compiler that expects text input. Once past the reader, everything is an in-memory representation, so we never have to get back to text. That said, it's trivial to get back to text if you want, because Lisps also include a printer which takes an in-memory representation and prints the glyphs that represent their readable (i.e., can be read by the reader) textual representation. 3. The evaluation model explicitly supports tagging some code as a macro to go through macro expansion before being evaluated "normally". 4. It is idiomatic to use macros to solve code re-use problems that cannot be readily handled by standard function calls.[1] These four Lisp traits are not exclusive to Lisp, necessarily. As I mentioned above, you could build a macro expansion and code generation library that is part of a build process for any compiled language. Eval, in interpreted languages, is as expressive as Lisp macros, but is incredibly poorly supported and unergonomic in comparison to Lisp macros. It's an ergonomic and feasibility thing more than a capability thing. That said, POSIX shell is turing complete, so you don't need anything like C# or Lisp or Haskell or anything else we've discussed to get your computation done. The reaction you might have to comparing shell scripting to Haskell is not dissimilar to the reaction an experienced Lisp programmer might have to comparing AST manipulation and code gen through a library to Lisp's macro facilities. [0]: Clojure, specifically abstracts the idea of callable, and there are more callable things than special operators, macros, and functions. [1]: Languages such as Haskell and OCaml continue to push the boundaries of what can be handled by "standard function calls" (which term I hesitate to use, given its imprecision). That said, given that macros allow the use of a turing-complete language to generate inputs to the evaluator, they allow you to implement any language construct that is computable without modifying the evaluation infrastructure.

greggyb 5y ago

As for REPL driven development, it's not about typing code at a REPL. This video provides a pretty good example of working at a REPL in Clojure. Note that with the exception of one doc reference at the beginning, all code is typed directly into a file buffer. The editor provides shortcuts to send various forms to the REPL for evaluation. https://vimeo.com/230220635 https://vimeo.com/230220635

p_l 5y ago

Consider that macros generally don't expand syntax - because there is very little syntax anyway, and what you have is a tree of data and function calls. A Macro just lets you write a function that will be executed by compiler to manipulate such tree, meaning that you can have equivalent of adding syntax like "with/using" from some languages by writing a function that takes as arguments the object and a block of code, then re-emits the same block of code but adding object.Dispose() call at the end. Or whatever else you need. But they are still normal functions - it's the time of execution that changes.

mwattsun 5y ago
I generally never used macros when I wrote C because the C macro language is lacking, there are a lot of pitfalls and saving function call overhead is usually not that important. I think I'll try to get a new understanding of Lisp macros by trying to come up with a language feature in C# or Dart that I really want but can't implement without a lot of non-intuitive difficulty, but could easily in a Lisp style macro that the compiler could use to build in a new feature at compile time.