7 ms·

I've learned Clojure. Tried and failed to see this unique power of macros. Truly asking for help: can you help explain what can I do with macros that I cannot

• deleted 5y ago
  [deleted]
• mwattsun 5y ago
  My impression, and I hope I'm wrong, is that macros and meta programming were powerful ideas 30 years ago, but now most modern languages have generics, template meta programming [1] and reflection. I'm a curious amateur and probably out to lunch, but there you go. [1] Some other languages support similar, if not more powerful, compile-time facilities (such as Lisp macros), but those are outside the scope of this article. https://en.wikipedia.org/wiki/Template_metaprogramming https://en.wikipedia.org/wiki/Template_metaprogramming
  • bcrosby95 5y ago
    The existence of template metaprogramming is an argument for macros, not against them. It shows the lengths to which programmers will abuse and misuse a feature so that they can reach their goals.
    • mwattsun 5y ago
      When I first started writing asm code, it was accepted to write self-modifying code (programmatically create new code, then jmp to it.) Then I discovered and was told it was a bad idea. So that's how I view Lisp macros - creating code on the fly and executing it. When I was studying this issue, I came across femtolisp, which is a standalone Lisp by the author of Julia, Jeff Bezanson. Julia has metaprogramming so I wanted to look at it. In this video [1] Professor Steve G. Johnson explains metaprogramming (around 10 min mark) and says "most of the time, don't do metaprogramming." He points out in the base Julia there are 36,798 methods, but only 138 macros and 14 generated functions (.04% of the code), so I have the impression it is still frowned upon. [1] https://www.youtube.com/watch?v=mSgXWpvQEHE https://www.youtube.com/watch?v=mSgXWpvQEHE
      • greggyb 5y ago
        > So that's how I view Lisp macros - creating code on the fly and executing it. This is an incorrect assumption. In compiled Lisps[0], macros are a compile-time construct, which manipulate the data structures that represent your code[1]. All manipulation occurs at compile-time. In interpreted Lisps, macro evaluation is temporally intertwined with program execution, but the two phases are logically distinct. [0]: Compilation time in a Lisp is not necessarily similar to compilation in other languages. Working at a REPL does not imply that interpretation is happening. Clojure is an example of a Lisp that is exclusively compiled. Even when sending individual expressions to the REPL, they are compiled before being evaluated. [1]: From a reply elsewhere in this thread: https://youtu.be/P76Vbsk_3J0?t=1333 https://youtu.be/P76Vbsk_3J0?t=1333 See this video for an overview of the core of Clojure. The link starts with an overview of Clojure's data types and data structures (skipping the intro for the reasons behind Clojure). This leads directly into the evaluation model, which is based on the data structures of Clojure. Watch through ~1h:30m to get a feel for what macros are used for in Clojure's core. This gives a very good overview of the evaluation model for macros; this is generally applicable to Lisp macros outside of Clojure as well, though there are nuances to different dialects. I recommend that you start at the linked time, but if you want to jump straight to the evaluation model, you can jump to ~47m.
        • mwattsun 5y ago
          Thank you for taking the time to answer! At compile time vs at runtime is an important distinction that I need to think about. I'll start the video from the beginning because I'm curious about Clojure and most things Rich Hickey does, since I liked is Simple Made Easy video so much.
        • mwattsun 5y ago
          So Clojure programs are data structures, and you can pass a structure to a macro to rewrite it into something the compiler can evaluate. In this way you can extend the language. You can build your own DSL that solves your particular problem in an organic way that grows as you go along. If I understand that correctly I don't get it, because I would still have to write the macro, which in C# I would write as a function that I call in my source code, which doesn't grow the syntax of C#, but grows the program to a DSL that solves my problem, albeit not as nicely as in a REPL. Maybe this is all about the REPL and I don't use them or see any appeal in them. I like to write my text, look at it, and hit run. You write a REPL line and it disappears. I'll keep learning. I'm sure the light bulb will go off eventually.
  • pandaman 5y ago
    Macros are not generics even though you can use them as such. Macros define new statements (and definitions for the languages that make a distinction). This allows you to extend the language to the specific domain and it's impossible to do with functions/generics in general case.
• reikonomusha 5y ago
  I can build a type system [0] with power approximately equal to that of Haskell within Lisp using macros. Not possible or even conceivable to do with functions. [0] https://coalton-lang.github.io/20211010-introducing-coalton/ https://coalton-lang.github.io/20211010-introducing-coalton/
• greggyb 5y ago
  From a Clojure perspective, here are a few things to think about and explore to help understand where the power and value of macros lie. 1. Clojure does not have all of its various conditionals as language built-ins. There is a single conditional construct built in to the language, if. The rest are all defined in terms of if. Function arguments are eagerly evaluated in Clojure. How do you write a short-circuiting function in Clojure? (when predicate consequent) as a function must evaluate both the predicate and the consequent. If consequent is slow to calculate then your whole when expression is as slow as consequent, even if predicate is false and we do not need consequent. when is a Clojure macro based on the if special form. 2. core.async is a mere library.[0] This implements CSP (same idea as Go's channels) and async programming (the sort of stuff that needs compiler support in other languages such as C# or Go) as a library. If Rich Hickey didn't write core.async, you could build this control flow yourself for async programming. In fact, you can create arbitrary control flow mechanisms using macros. Common Lisp's entire object system can be implemented in macros. 3. The only special forms in Clojure are def, if, fn, let, loop, recur, do, new, ., throw, try, set!, quote, and var. The rest of what you would consider the core of Clojure consists of functions and macros. Much of the control flow functionality is implemented in macros.[1] The core of macros is that they allow you to evaluate arbitrary user code at compile time. Clojure code is represented as native Clojure data structures (the same is true for all Lisps). The entire standard library which you are accustomed to using to manipulate standard data structures is available at compile time to transform the data structures that represent your code. I can't explain this better than Rich Hickey does in [1], so I encourage you to watch the linked section of that talk. [0]: https://youtu.be/yJxFPoxqzWE?t=560 https://youtu.be/yJxFPoxqzWE?t=560 See this video for an overview of core.async. Later on he talks about the stuff that is implemented as macros in the library. This link starts up when he's talking about C# style async. [1]: https://youtu.be/P76Vbsk_3J0?t=1333 https://youtu.be/P76Vbsk_3J0?t=1333 See this video for an overview of the core of Clojure. The link starts with an overview of Clojure's data types and data structures (skipping the intro for the reasons behind Clojure). This leads directly into the evaluation model, which is based on the data structures of Clojure. Watch through ~1h:30m to get an overview of what macros are used for in Clojure's core.
• Zak 5y ago
  Here's a simple example you can't write in most languages. (first-working (get-it-from-the-cache) (get-it-from-the-database) (get-it-from-an-external-api) (compute-it-the-slow-way) default-value) Note each of those functions could throw an exception. You want to ignore it (or you could log it) and move on to the next.
  • atdt 5y ago
    Maybe I am missing something. Here's how I might write something equivalent in Python: for f in GetFromCache, GetFromDatabase, GetFromAPI, Compute: try: return f() except Exception as ex: log(ex) return default_value (You'd probably define a custom exception class for your application that represents the kinds of errors you can tolerate, so you don't end up swallowing programming errors, but you get the idea.)
    • roenxi 5y ago
      Mmm. You've given a code snippet while Zak had a single macro call. Let's make your code into a function with signature: def atdt_exception_suppression(*list_of_fns): ... What you've done is leverage the fact that functions are first class objects in Python (ie, can be passed as arguments) to achieve the same result as a macro. Which is cool, it works. Probably slightly better than a macro for this use case because we get a neat function boundary. But macros are a powerful option because it doesn't matter if functions are first class or not. If you are using a language where functions aren't first class objects, a macro system would let Zak implement anyway. So you are correctly identifying that Python can implement this in an alternate way as a function, but a macro system is sufficient to do this and a whole batch of other tricks even if first class functions aren't available as a feature.
    • cnasc 5y ago
      You’re running through a list of functions, the lisp example has a list of _function calls_ (which are not evaluated until necessary). Consider how your approach would have to change if each of those functions takes different parameters.
    • 5y ago
• roenxi 5y ago
  Well, for Clojure ... Spend 5 minutes playing with -> [0]. Appreciate how pretty it makes the code and how nice it is for laying out sequential operations. What a fun operator. Then try to implement it yourself as a function. When you discover that you can't, you will have discovered the unique power of macros (note the docs link to source code so you can see the readable 8-line macro implementation). You've also learned a very practical skill, which is how to write threading macros that handle your special edge case fail conditions. It comes in handy from time to time. [0] https://clojure.github.io/clojure/clojure.core-api.html#clojure.core/-%3E https://clojure.github.io/clojure/clojure.core-api.html#cloj...
  • grumpyprole 5y ago
    Note that this cannot be written as a function only because Clojure uses strict (call-by-value) evaluation everywhere and has no way to change this (other than Macros). In theory languages could provide annotations to allow a user to specify whether parameters are call-by-value or call-by-need. Haskell and Scala both provide this. But I don't think this is a bad example at all, I tend to think of macros as "extending the language/compiler", since they have access to the abstract syntax tree of their inputs, which is what is being done here.
    • roenxi 5y ago
      Yeah. And for the interested reader, this is unique power of macros. It doesn't matter what specific combination of language features are supported. Macros mean you can implement things anyway. It doesn't really matter what the evaluation logic of the language is, a macro can create syntax that does what you want it to. If there is a macro system available, a new operator can be implemented in any language. If there is no macro system, it depends what features the language supports.
• davidrupp 5y ago
  Colloquially: You can rewrite source code into a different form as it's being evaluated, then have it be further evaluated as if it were originally written in the new form. One fairly simple example is Clojure's `when`, which is an expression that starts off looking like (when (= 0 1) (+ 1 2)) and gets re-written to (if (= 0 1) (do (+ 1 2))) before being further compiled. This rewriting happens as many times as necessary until an expression "bottoms out" and can be evaluated in terms of only built-ins. It's instructive to see how much of e.g., Clojure is implemented as macros. https://github.com/clojure/clojure/blob/master/src/clj/clojure/core.clj#L495 https://github.com/clojure/clojure/blob/master/src/clj/cloju... shows pretty clearly the correspondence between the original form and the rewritten form in the definition of `when`. `defn`, which is so foundational to idiomatic Clojure that it's easy to mistake it for a built-in, is a much more complicated example.
• beders 5y ago
  Macros give you control over evaluation order. Fns don't. (or a b) If a is truthy, b will never evaluate
  • grumpyprole 5y ago
    Note that functions in other languages, like Haskell and Scala, do allow some control over evaluation order. One can annotate function parameters to achieve call-by-value or call-by-need as required. Macros can analyse/manipulate the abstract syntax tree of their inputs, so I think of them as extending the compiler.
  • lispm 5y ago
    (or (lambda () a) (lambda () b)) Now you only need to define the function OR...
• phoe-krk 5y ago
  Lack of macros makes nothing impossible, but also makes many things harder. I think of them as code generators, tiny compilers on a micro scale; they are invoked by the proper compiler during compilation phase, as opposed to runtime, allowing you to inject code that you'd otherwise need to inline by hand. It's possible to write pretty complex things like control flow abstractions using just structures and functions/methods, e.g. a CL-esque condition system in Java which only uses classes, static methods, and Java lambdas[0] for its syntax. Possible, but also IMO ugly when compared to the CL counterpart, because the low-level but irrelevant details (such as instantiation via `new` or generics) are still presented to the programmer. [0] https://github.com/phoe/cafe-latte https://github.com/phoe/cafe-latte