8 ms·

What curious newbies might miss from discussions of macros is that they are a separate language operating at compile time. They are usually advertised as code o

What curious newbies might miss from discussions of macros is that they are a separate language operating at compile time. They are usually advertised as code operating on itself or something like that. They're not. Powerful, yes, but not that cool.

procparam 7y ago
A separate language? How do you mean? This is one of the big advantages of Lisp macros over C macros. In Lisp you write macros using Lisp, including normal user-defined functions. In C, on the other hand, you write macros using "C Preprocessor directives"
- Rerarom 7y ago
  Yes, of course, I get that it has the same syntax, which is not the case with C macros and which makes macros way better. Thing is, there is one Lisp that executed at compile time which expands the macros, and then there is the resulting code which executed at runtime. It's not self-modifying code. It's just code that operates on some other code, written in the same syntax.
  - btilly 7y ago
    Whatever artificial division you believe exists between "compile time" and "run time" in Lisp is almost certainly your misunderstanding, and does not reflect how the language actually works.
    - threatofrain 7y ago
      Perhaps I could use some clarity on this matter too, and hopefully with a bit of gentleness. As I understand it, some Lisps do have an explicit compile phase and there is discussion among the language users over the runtime cost of macro abstractions. https://docs.racket-lang.org/guide/stx-phases.html https://docs.racket-lang.org/guide/stx-phases.html I understand there may be some philosophical or theoretical interest in framing Lisp the way you do, but does that invalidate the close-to-the-app thinking about the costs of macros and where those costs are distributed?
      - ludston 7y ago
        I'm not so familiar with scheme, but I can perhaps give useful answer regarding how this works with SBCL (Probably the most well known common lisp implementation). In SBCL, we would commonly say that code is "evaluated" rather than compiled or interpreted. This is because "compiler" is referring specifically to when assembly is emitted and "interpreter" is referring to when code is being used to call already compiled code. In the case of SBCL, it is doing both of these things simultaneously when it is reading a ".lisp" file. For example, if this is the contents of my .lisp file: (defun boop () (print "hello")) (boop) ;; prints hello (defun scoop () (print "goodbye")) The boop function is being evaluated (and therefore compiled), and then called on the next line. In traditional algol based languages like c and java with a separate compilation phase, it would not be possible to call boop before scoop has been turned into machine code. So you are asking how the costs of evaluating lisp are distributed? For one, the assembler generated whilst evaluating the .lisp file can be cached in a .fasl file so that it does not need to be evaluated again to be loaded into other projects. Another is that macro's can be evaluated once during function definition. e.g. (defmacro moob () `(print "kellog")) (defun foo () (moob)) ; evaluates to (defun funky () (print "kellog")) (defmacro moob () `(print "fish")) ; redefining moob (foo) ; will print "kellog" because funky was evaluated before moob was redefined. In this case, the cost of calling moob is trivial because it only occurs once during the definition of foo. It will slow down the initial load of your program, but no subsequent calls to foo.
        lispm 7y ago
        > This is because "compiler" is referring specifically to when assembly is emitted SBCL has both an in-core compiler (compiles to machine code in memory) and a file compiler (compiles to machine code in an file). SBCL also has an interpreter (relatively new and which usually is not used), which executes Lisp source as data.
        omaranto 7y ago
        I didn't know SBCL had an interpreter! (For those who didn't know either, see [1].) When would you use it? [1] http://www.sbcl.org/manual/#Interpreter http://www.sbcl.org/manual/#Interpreter
  - jlarocco 7y ago
    That's not true, though. When running in the REPL (the most common way to interface with Lisp) the compile time and runtime Lisps are the same. For example, here's a copy/paste from a REPL session that defines a macro that defines a function. The macro (at "compile time") prints information about the function (just its argument count) to stdout before using defun (itself a macro) to actually define the function. Next I call the new function and print out a disassembly, just to show the function is in fact compiled "CL-USER> " is the prompt in the REPL I use: CL-USER> (defmacro my-defun (name arguments &body body) (format t "Defining ~a, taking ~a arguments~%" name (length arguments)) `(defun ,name ,arguments ,@body)) MY-DEFUN CL-USER> (my-defun omg-2 (value) (* value value)) Defining OMG-2, taking 1 arguments OMG-2 CL-USER> (omg-2 34) 1156 CL-USER> (disassemble #'omg-2) ; disassembly for OMG-2 ; Size: 33 bytes. Origin: #x52ED2714 ; OMG-2 ; 14: 498B5D10 MOV RBX, [R13+16] ; thread.binding-stack-pointer ; 18: 48895DF8 MOV [RBP-8], RBX ; 1C: 488BD6 MOV RDX, RSI ; 1F: 488BFE MOV RDI, RSI ; 22: FF1425C0001052 CALL QWORD PTR [#x521000C0] ; GENERIC-* ; 29: 488B75F0 MOV RSI, [RBP-16] ; 2D: 488BE5 MOV RSP, RBP ; 30: F8 CLC ; 31: 5D POP RBP ; 32: C3 RET ; 33: CC10 INT3 16 ; Invalid argument count trap NIL CL-USER> It is also possible to compile a Lisp program to an executable (or byte code or whatever) and run it, and not use the dynamic aspect of it.
    - lispm 7y ago
      The code gets compiled (and macro expanded), before it is running. That's a definition of compile-time in Lisp. In an interpreter version of Lisp, the interpreter may expand the macros at runtime.
- ImprobableTruth 7y ago
  I wouldn't say they are a seperate language, but they definitely feel like a seperate layer. They're not first class objects like functions (so you can't pass them to functions) and you can't evaluate the arguments (since macros are only directly present during compilation/interpretation), it's purely a mapping from source code to source code. For example, you can't rewrite a list based on it's values or form. You can rewrite the list to a program that does the transformation when run, but the seperation feels very tangible to me. Something like FEXPRs would be closer to being "just the same as normal code", but there's good reason why macros are the way they are.
  - samatman 7y ago
    You can absolutely pass a macro to a function. It's a list! What you can't do is use funcall or apply on it, because it isn't a function. What you do is, you use macroexpand-1, like so: https://stackoverflow.com/questions/50754347/macro-with-a-list-of-macros-as-argument-in-common-lisp https://stackoverflow.com/questions/50754347/macro-with-a-li...
btilly 7y ago
Your comment is true about most languages, but false about Lisp. Lisp has no compile time vs run time distinction. It only has lists. What is done with a list depends on what the first thing in the list is determined to be: 1. Function - evaluate the arguments then pass it to the function. 2. Macro - manipulate the list according to the macro and then get another list. 3. Special Form - follow the special rules for the special form. (The list of special forms varies from Lisp to Lisp, and are the elementary building blocks from which the language was built.) In Lisp, macros are advertised as code operating on itself, because that is exactly what they are. And yes, they are exactly that cool.
- Rerarom 7y ago
  Every reference on Lisp macros says that they are executed at compile-time.
  - btilly 7y ago
    From http://www.paulgraham.com/icad.html http://www.paulgraham.com/icad.html: There is no real distinction between read-time, compile-time, and runtime. You can compile or run code while reading, read or run code while compiling, and read or compile code at runtime. That is from one of the founders of YCombinator, the author of http://www.paulgraham.com/onlisptext.html http://www.paulgraham.com/onlisptext.html and http://www.paulgraham.com/acl.html http://www.paulgraham.com/acl.html.
    - Rerarom 7y ago
      That's the kind of misleading text I was talking about in my first comment.
      - ScottBurson 7y ago
        There is a conceptual distinction between read time, compile time, and run time, but it's only conceptual: all these things happen in a single process in an interleaved manner. When you type an expression into the REPL, first it gets read, then it gets compiled to some extent that depends on the implementation, then it gets run. Some implementations run every expression you type through the full native-code compiler, but others just interpret the expression, expanding any macros as they encounter them. In the latter case, it's switching back and forth between "run time" and "compile time" potentially many times during the interpretation of a single expression. It is certainly true that macros are a different kind of code from most ordinary code, because what they're doing is different. But importantly, they are written in the same Turing-complete language. I once wrote a program (a C implementation for Lisp Machines) that allowed the user to interactively execute C expressions. Obviously, it had to parse the C code according to the syntax of that language, but the parser produced list structure that consisted entirely of macro calls. The guts of the compiler were implemented as a collection of macros that translated the output of the parser into Lisp Machine Lisp. These macros did things like accessing and updating a symbol table — far beyond what you could do in a C macro.
ScottBurson 7y ago
I don't know what your threshold for "cool" is, but an example of something that's straightforward in Lisp is if you want to generate a hierarchy of class declarations from a schema, so the classes have certain instance variables and methods defined in some regular way; for instance, you might want to auto-generate a method for each class that returns a collection of the values of certain of its instance variables. In most any language other than Lisp, you'd have to define your own schema syntax and write a parser for it and a generator that outputs the declarations in the target language. In a nutshell, it would probably be painful enough that you wouldn't bother; you'd just type in all the boilerplate manually. In Lisp, not only is writing such a schema translator relatively straightforward, but you get incremental recompilation for free.

procparam 7y ago

A separate language? How do you mean? This is one of the big advantages of Lisp macros over C macros. In Lisp you write macros using Lisp, including normal user-defined functions. In C, on the other hand, you write macros using "C Preprocessor directives"

Rerarom 7y ago
Yes, of course, I get that it has the same syntax, which is not the case with C macros and which makes macros way better. Thing is, there is one Lisp that executed at compile time which expands the macros, and then there is the resulting code which executed at runtime. It's not self-modifying code. It's just code that operates on some other code, written in the same syntax.
- btilly 7y ago
  Whatever artificial division you believe exists between "compile time" and "run time" in Lisp is almost certainly your misunderstanding, and does not reflect how the language actually works.
  - threatofrain 7y ago
    Perhaps I could use some clarity on this matter too, and hopefully with a bit of gentleness. As I understand it, some Lisps do have an explicit compile phase and there is discussion among the language users over the runtime cost of macro abstractions. https://docs.racket-lang.org/guide/stx-phases.html https://docs.racket-lang.org/guide/stx-phases.html I understand there may be some philosophical or theoretical interest in framing Lisp the way you do, but does that invalidate the close-to-the-app thinking about the costs of macros and where those costs are distributed?
    - ludston 7y ago
      I'm not so familiar with scheme, but I can perhaps give useful answer regarding how this works with SBCL (Probably the most well known common lisp implementation). In SBCL, we would commonly say that code is "evaluated" rather than compiled or interpreted. This is because "compiler" is referring specifically to when assembly is emitted and "interpreter" is referring to when code is being used to call already compiled code. In the case of SBCL, it is doing both of these things simultaneously when it is reading a ".lisp" file. For example, if this is the contents of my .lisp file: (defun boop () (print "hello")) (boop) ;; prints hello (defun scoop () (print "goodbye")) The boop function is being evaluated (and therefore compiled), and then called on the next line. In traditional algol based languages like c and java with a separate compilation phase, it would not be possible to call boop before scoop has been turned into machine code. So you are asking how the costs of evaluating lisp are distributed? For one, the assembler generated whilst evaluating the .lisp file can be cached in a .fasl file so that it does not need to be evaluated again to be loaded into other projects. Another is that macro's can be evaluated once during function definition. e.g. (defmacro moob () `(print "kellog")) (defun foo () (moob)) ; evaluates to (defun funky () (print "kellog")) (defmacro moob () `(print "fish")) ; redefining moob (foo) ; will print "kellog" because funky was evaluated before moob was redefined. In this case, the cost of calling moob is trivial because it only occurs once during the definition of foo. It will slow down the initial load of your program, but no subsequent calls to foo.
      - lispm 7y ago
        > This is because "compiler" is referring specifically to when assembly is emitted SBCL has both an in-core compiler (compiles to machine code in memory) and a file compiler (compiles to machine code in an file). SBCL also has an interpreter (relatively new and which usually is not used), which executes Lisp source as data.
        omaranto 7y ago
        I didn't know SBCL had an interpreter! (For those who didn't know either, see [1].) When would you use it? [1] http://www.sbcl.org/manual/#Interpreter http://www.sbcl.org/manual/#Interpreter
- jlarocco 7y ago
  That's not true, though. When running in the REPL (the most common way to interface with Lisp) the compile time and runtime Lisps are the same. For example, here's a copy/paste from a REPL session that defines a macro that defines a function. The macro (at "compile time") prints information about the function (just its argument count) to stdout before using defun (itself a macro) to actually define the function. Next I call the new function and print out a disassembly, just to show the function is in fact compiled "CL-USER> " is the prompt in the REPL I use: CL-USER> (defmacro my-defun (name arguments &body body) (format t "Defining ~a, taking ~a arguments~%" name (length arguments)) `(defun ,name ,arguments ,@body)) MY-DEFUN CL-USER> (my-defun omg-2 (value) (* value value)) Defining OMG-2, taking 1 arguments OMG-2 CL-USER> (omg-2 34) 1156 CL-USER> (disassemble #'omg-2) ; disassembly for OMG-2 ; Size: 33 bytes. Origin: #x52ED2714 ; OMG-2 ; 14: 498B5D10 MOV RBX, [R13+16] ; thread.binding-stack-pointer ; 18: 48895DF8 MOV [RBP-8], RBX ; 1C: 488BD6 MOV RDX, RSI ; 1F: 488BFE MOV RDI, RSI ; 22: FF1425C0001052 CALL QWORD PTR [#x521000C0] ; GENERIC-* ; 29: 488B75F0 MOV RSI, [RBP-16] ; 2D: 488BE5 MOV RSP, RBP ; 30: F8 CLC ; 31: 5D POP RBP ; 32: C3 RET ; 33: CC10 INT3 16 ; Invalid argument count trap NIL CL-USER> It is also possible to compile a Lisp program to an executable (or byte code or whatever) and run it, and not use the dynamic aspect of it.
  - lispm 7y ago
    The code gets compiled (and macro expanded), before it is running. That's a definition of compile-time in Lisp. In an interpreter version of Lisp, the interpreter may expand the macros at runtime.
ImprobableTruth 7y ago
I wouldn't say they are a seperate language, but they definitely feel like a seperate layer. They're not first class objects like functions (so you can't pass them to functions) and you can't evaluate the arguments (since macros are only directly present during compilation/interpretation), it's purely a mapping from source code to source code. For example, you can't rewrite a list based on it's values or form. You can rewrite the list to a program that does the transformation when run, but the seperation feels very tangible to me. Something like FEXPRs would be closer to being "just the same as normal code", but there's good reason why macros are the way they are.
- samatman 7y ago
  You can absolutely pass a macro to a function. It's a list! What you can't do is use funcall or apply on it, because it isn't a function. What you do is, you use macroexpand-1, like so: https://stackoverflow.com/questions/50754347/macro-with-a-list-of-macros-as-argument-in-common-lisp https://stackoverflow.com/questions/50754347/macro-with-a-li...

btilly 7y ago

Your comment is true about most languages, but false about Lisp. Lisp has no compile time vs run time distinction. It only has lists. What is done with a list depends on what the first thing in the list is determined to be: 1. Function - evaluate the arguments then pass it to the function. 2. Macro - manipulate the list according to the macro and then get another list. 3. Special Form - follow the special rules for the special form. (The list of special forms varies from Lisp to Lisp, and are the elementary building blocks from which the language was built.) In Lisp, macros are advertised as code operating on itself, because that is exactly what they are. And yes, they are exactly that cool.

Rerarom 7y ago
Every reference on Lisp macros says that they are executed at compile-time.
- btilly 7y ago
  From http://www.paulgraham.com/icad.html http://www.paulgraham.com/icad.html: There is no real distinction between read-time, compile-time, and runtime. You can compile or run code while reading, read or run code while compiling, and read or compile code at runtime. That is from one of the founders of YCombinator, the author of http://www.paulgraham.com/onlisptext.html http://www.paulgraham.com/onlisptext.html and http://www.paulgraham.com/acl.html http://www.paulgraham.com/acl.html.
  - Rerarom 7y ago
    That's the kind of misleading text I was talking about in my first comment.
    - ScottBurson 7y ago
      There is a conceptual distinction between read time, compile time, and run time, but it's only conceptual: all these things happen in a single process in an interleaved manner. When you type an expression into the REPL, first it gets read, then it gets compiled to some extent that depends on the implementation, then it gets run. Some implementations run every expression you type through the full native-code compiler, but others just interpret the expression, expanding any macros as they encounter them. In the latter case, it's switching back and forth between "run time" and "compile time" potentially many times during the interpretation of a single expression. It is certainly true that macros are a different kind of code from most ordinary code, because what they're doing is different. But importantly, they are written in the same Turing-complete language. I once wrote a program (a C implementation for Lisp Machines) that allowed the user to interactively execute C expressions. Obviously, it had to parse the C code according to the syntax of that language, but the parser produced list structure that consisted entirely of macro calls. The guts of the compiler were implemented as a collection of macros that translated the output of the parser into Lisp Machine Lisp. These macros did things like accessing and updating a symbol table — far beyond what you could do in a C macro.

ScottBurson 7y ago

I don't know what your threshold for "cool" is, but an example of something that's straightforward in Lisp is if you want to generate a hierarchy of class declarations from a schema, so the classes have certain instance variables and methods defined in some regular way; for instance, you might want to auto-generate a method for each class that returns a collection of the values of certain of its instance variables. In most any language other than Lisp, you'd have to define your own schema syntax and write a parser for it and a generator that outputs the declarations in the target language. In a nutshell, it would probably be painful enough that you wouldn't bother; you'd just type in all the boilerplate manually. In Lisp, not only is writing such a schema translator relatively straightforward, but you get incremental recompilation for free.