4 ms·

What are lisp macros for if not to delay evaluation? I assume you're making a distinction between purpose and mechanism here? I've always understood the semant

What are lisp macros for if not to delay evaluation? I assume you're making a distinction between purpose and mechanism here? I've always understood the semantics to be: fn: (a b c) => (call a (eval b) (eval c)) mac: (a b c) => (eval (call a b c))

lispm 8y ago
That might be more like what was called an FEXPR mechanism in some early Lisps or some niche Lisps, or even in languages like R. https://en.wikipedia.org/wiki/Fexpr https://en.wikipedia.org/wiki/Fexpr When an FEXPR is called, it gets unevaluated args and can then at runtime decide what to do. Macros OTOH are a different mechanism, where expressions get rewritten at macro expansion time, which can be for example at compile time. The macro then gets called with arbitrary expressions it encloses (those don't need to be valid code by themselves) and computes new source. Thus macros are code generators from expressions. In a compiled implementation, macro expansion is done with compilation interleaved: each form gets expanded, even repeatedly until it no longer expands into a macro, and then the resulting non-macro form gets compiled. Thus in a way a macro does not delay execution, it does the opposite: it actually shifts computation to compile time -> the computation of code from expressions and the computation of arbitrary side effects in the compile-time environment. In an interpreter version of Lisp, the macro gets also expanded at runtime - but in its own macroexpansion during evaluation. There eval will call the macroexpander repeatedly until it gets a non-macro form. Now, what can you do with arbitrary complex code generators at compile time? https://stackoverflow.com/a/2563308/69545 https://stackoverflow.com/a/2563308/69545 Actually Paul Graham wrote a classical Lisp book explaining a bunch of things around macros. Available here for download: http://www.paulgraham.com/onlisp.html http://www.paulgraham.com/onlisp.html The classic Lisp article which motivated the move from FEXPRs to macros: https://www.nhplace.com/kent/Papers/Special-Forms.html https://www.nhplace.com/kent/Papers/Special-Forms.html
- reitzensteinm 8y ago
  I don't think what I wrote is a FEXPR, which doesn't evaluate the result of the function call. I'm having a hard time parsing what you wrote or any of the sources you linked in such a way that says the semantics of macros differs from what I wrote (certainly the implementation gets a lot more complex and there are subtleties).
  - lispm 8y ago
    That's why I wrote 'more like'. Given that you haven't defined any semantics of your operators, it's more like a guess. Not sure if this helps you. But let's define a macro A: CL-USER 32 > (defmacro a (b c) (print (list :macro-expansion b c)) (list 'print (list 'quote (list :runtime :b b :c c)))) A This macro does two things: it prints something at macro expansion time and then generates some code it returns as a value. Now we can use this macro in some code: CL-USER 33 > (defun test () (a 21 42)) TEST If we now compile the function, the macro gets executed and prints something: CL-USER 34 > (compile 'test) (:MACRO-EXPANSION 21 42) TEST NIL NIL We can also call the macroexpander independent of the compiler. The we a) get the side effect of the print statement and we see the result: CL-USER 35 > (macroexpand '(a 20 30)) (:MACRO-EXPANSION 20 30) (PRINT (QUOTE (:RUNTIME :B 20 :C 30))) T At runtime we call the test function: CL-USER 36 > (test) (:RUNTIME :B 21 :C 42) ; <- the printed output (:RUNTIME :B 21 :C 42) ; print also returns its arg Thus all the macro expansions have been done at compile time and we have generated some code there. No macro expansion at runtime. Thus it has to do with code generation and code execution at macro expansion - nothing about 'delaying' something. The example isn't useful, but imagine a macro INFIX (infix a * b + c) which rewrites the expression to the Lisp expression: (+ (* a b) c) There is nothing about 'delaying' -> it's just rewriting the form. Ideally at compile time. There are many other examples which do something different.
    - reitzensteinm 8y ago
      But an infix transform does delay the evaluation of expressions you've passed to it. Eg (infix (avg b) * (sgn a)). And most practical uses of macros involve passing in expressions that will be later evaluated verbatim, e.g. (time (reduce + (range 100))) I was only intending responding to this: "That's a common misconception that Lisp macros are mostly used to 'delay' evaluation." Which just seems to me confuses the matter in response to the article that was posted. At a very low level, macros operate by having the evaluation of their arguments delayed. At a higher level, they're used to implement sugared forms which usually contain valid code (though not necessarily code that makes sense in isolation). I know you're very experienced with lisp, and I'm not trying to correct you technically, but to me this stuff only clicked when implementing a metacircular evaluator and learning when and how to perform evaluation. The blog post to me reads like the author was going through a similar process.
- thesz 8y ago
  https://www.quotes.net/quote/36312 https://www.quotes.net/quote/36312 "The primary purpose of the DATA statement is to give names to constants; instead of referring to pi as 3.141592653589793 at every appearance, the variable PI can be given that value with a DATA statement and used instead of the longer form of the constant. This also simplifies modifying the program, should the value of PI change." Or, quoting the other aphorism, what is constant for someone is a variable for someone else. What is run time for LISP (which lacked good compiler at early stages - the famous two horse asses width constraint) can freely be a compile time for other language. Or for a LISP, for that matter, in our time. The compile time expansion allows for better (because faster) error checking.
abecedarius 8y ago
Take a macro like (infix x + vx * dt). It's not equivalent to calling some function with thunked arguments -- it has to analyze the literal text of the arguments to turn this into (+ x (* vx dt)).
jfoutz 8y ago
i'll defer to lispm's expert opinion, but i always thought of macros as _conditional_ evaluation. `if` is a special form because it does not evaluate the branch not taken. it's tricky to implement `if` without using `if` or `cond`. macros make it easy to implement your own special forms. in your example, you might want to implement `first`. (first b c) -> (eval b) i can't think of a way (but it might be possible) to not evaluate c with a regular function. first is trivial and dumb, but it's obvious how to turn that into short circuiting `and` or `or`.
- bitwize 8y ago
  Macros are more general than that; they are generalized expression rewriters. `if` is tricky to implement in general because you need a primitive to select an alternative based on whether a given value is truthy or falsy. Let's pretend we have such a primitive that's not a special form called `call-if`. Then you can implement `if` in terms of `call-if` by writing it as a macro that transforms this: (if (condition? x) (do-a x) (do-b x)) into this: (call-if (condition? x) (lambda () (do-a x)) (lambda () (do-b x))) (The semantics of `call-if`, by the way, are exactly the semantics exposed by the Smalltalk ifTrue:ifFalse: family of methods.) But macros can do more than merely wrap things in lambdas, which alone is sufficient for delayed and conditional evaluation. They can quote identifiers that are passed into them, allowing you to write new defining forms and even introduce new bindings. A nontrivial, but understandable, example is the `define-record-type` special form from SRFI 9, which can define a new disjoint record type and a constructor, predicate, and field accessors for that type all in one form. If you are willing to break hygiene and have `defmacro` or something like `syntax-case` available, you can even introduce new identifiers from out of nowhere, and write something like Common Lisp's `defstruct` which lets you define a type and automatically derive names for the constructor, slot accessors, etc. from the names of the type and slots. Macros are a tremendously powerful, general code generation and rewriting tool, built right into the language. Underestimate them at your own peril!
  - jfoutz 8y ago
    aww, now i'm embarrassed about my lack of imagination. I have to admit i forgot about the magic you can do with define. Excellent point. i kinda sorta think defun in common lisp injects the new function at top level regardless, but in scheme it's scoped. So in CL you could probably get away with a simple lambda (or function) to make your struct syntax, scheme would need the macro to put all of that stuff at the top level. Yeah, macros are crazy powerful
kazinator 8y ago
Macros determine what requirements apply to the processing of syntax. Which parts of that syntax, if any, are evaluated according to what rules, in what environment.
zimablue 8y ago
I think it's "'delayed evaluation' + 'transparent internal structure'". So delayed evaluation isn't enough, because the only thing you can do with N closures is to call them in any order, you can't create a complex new expression dependent on the substructure of the closures. The closest thing to a macro in (eg. python) imo looks like this. Instead of d = c * (a + b) class Adder: def __init__(self, left, right): self.left=left self.right = right class Multiplier: def __init__(self, left, right): self.left = left self.right = right d = Multiplier(Adder(a, b), c) Fill out the __call__ function and/or create an evaluator that recursively calls each class instance with it's parameters and you've made a system that works a lot like a lisp with a macros. Now you can do broadly /anything/, eg. replace every instance of an adder in the call tree with an adder+1. That's using the second required element of my description of macros- the complex substructure, to perform arbitrary transforms. You can't do that with just closures, unless they make their bound parameters accessible (and normally they don't directly, in python they do because it's absurdly dynamic). Of course this is not at compile-time, it's at "delayed execution time", but the difference is kind of slight imo. The benefit of a real lisp over doing this in python is that you don't need to rewrite every fn, operator, control flow statement into your own abstracted level by hand, and it's much faster. Also you can complete the loop and trivially write it back down into eval-able source code text.
kd0amg 8y ago
What are lisp macros for if not to delay evaluation? One class of use cases not covered in the blog post is that a variable appearing as a function's argument can only be a bound occurrence, a use of some name already in scope. A variable appearing as a macro's argument can be the binding occurrence of that name. This is why (using Racket examples) match, define-struct, for/list, and such things aren't functions. [1]: https://docs.racket-lang.org/reference/match.html?q=match#%28form._%28%28lib._racket%2Fmatch..rkt%29._match%29%29 https://docs.racket-lang.org/reference/match.html?q=match#%2... [2]: https://docs.racket-lang.org/reference/define-struct.html?q=define-struct#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define-struct%29%29 https://docs.racket-lang.org/reference/define-struct.html?q=... [3]: https://docs.racket-lang.org/reference/for.html?q=for%2Flist#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29 https://docs.racket-lang.org/reference/for.html?q=for%2Flist...

lispm 8y ago

That might be more like what was called an FEXPR mechanism in some early Lisps or some niche Lisps, or even in languages like R. https://en.wikipedia.org/wiki/Fexpr https://en.wikipedia.org/wiki/Fexpr When an FEXPR is called, it gets unevaluated args and can then at runtime decide what to do. Macros OTOH are a different mechanism, where expressions get rewritten at macro expansion time, which can be for example at compile time. The macro then gets called with arbitrary expressions it encloses (those don't need to be valid code by themselves) and computes new source. Thus macros are code generators from expressions. In a compiled implementation, macro expansion is done with compilation interleaved: each form gets expanded, even repeatedly until it no longer expands into a macro, and then the resulting non-macro form gets compiled. Thus in a way a macro does not delay execution, it does the opposite: it actually shifts computation to compile time -> the computation of code from expressions and the computation of arbitrary side effects in the compile-time environment. In an interpreter version of Lisp, the macro gets also expanded at runtime - but in its own macroexpansion during evaluation. There eval will call the macroexpander repeatedly until it gets a non-macro form. Now, what can you do with arbitrary complex code generators at compile time? https://stackoverflow.com/a/2563308/69545 https://stackoverflow.com/a/2563308/69545 Actually Paul Graham wrote a classical Lisp book explaining a bunch of things around macros. Available here for download: http://www.paulgraham.com/onlisp.html http://www.paulgraham.com/onlisp.html The classic Lisp article which motivated the move from FEXPRs to macros: https://www.nhplace.com/kent/Papers/Special-Forms.html https://www.nhplace.com/kent/Papers/Special-Forms.html

reitzensteinm 8y ago
I don't think what I wrote is a FEXPR, which doesn't evaluate the result of the function call. I'm having a hard time parsing what you wrote or any of the sources you linked in such a way that says the semantics of macros differs from what I wrote (certainly the implementation gets a lot more complex and there are subtleties).
- lispm 8y ago
  That's why I wrote 'more like'. Given that you haven't defined any semantics of your operators, it's more like a guess. Not sure if this helps you. But let's define a macro A: CL-USER 32 > (defmacro a (b c) (print (list :macro-expansion b c)) (list 'print (list 'quote (list :runtime :b b :c c)))) A This macro does two things: it prints something at macro expansion time and then generates some code it returns as a value. Now we can use this macro in some code: CL-USER 33 > (defun test () (a 21 42)) TEST If we now compile the function, the macro gets executed and prints something: CL-USER 34 > (compile 'test) (:MACRO-EXPANSION 21 42) TEST NIL NIL We can also call the macroexpander independent of the compiler. The we a) get the side effect of the print statement and we see the result: CL-USER 35 > (macroexpand '(a 20 30)) (:MACRO-EXPANSION 20 30) (PRINT (QUOTE (:RUNTIME :B 20 :C 30))) T At runtime we call the test function: CL-USER 36 > (test) (:RUNTIME :B 21 :C 42) ; <- the printed output (:RUNTIME :B 21 :C 42) ; print also returns its arg Thus all the macro expansions have been done at compile time and we have generated some code there. No macro expansion at runtime. Thus it has to do with code generation and code execution at macro expansion - nothing about 'delaying' something. The example isn't useful, but imagine a macro INFIX (infix a * b + c) which rewrites the expression to the Lisp expression: (+ (* a b) c) There is nothing about 'delaying' -> it's just rewriting the form. Ideally at compile time. There are many other examples which do something different.
  - reitzensteinm 8y ago
    But an infix transform does delay the evaluation of expressions you've passed to it. Eg (infix (avg b) * (sgn a)). And most practical uses of macros involve passing in expressions that will be later evaluated verbatim, e.g. (time (reduce + (range 100))) I was only intending responding to this: "That's a common misconception that Lisp macros are mostly used to 'delay' evaluation." Which just seems to me confuses the matter in response to the article that was posted. At a very low level, macros operate by having the evaluation of their arguments delayed. At a higher level, they're used to implement sugared forms which usually contain valid code (though not necessarily code that makes sense in isolation). I know you're very experienced with lisp, and I'm not trying to correct you technically, but to me this stuff only clicked when implementing a metacircular evaluator and learning when and how to perform evaluation. The blog post to me reads like the author was going through a similar process.
thesz 8y ago
https://www.quotes.net/quote/36312 https://www.quotes.net/quote/36312 "The primary purpose of the DATA statement is to give names to constants; instead of referring to pi as 3.141592653589793 at every appearance, the variable PI can be given that value with a DATA statement and used instead of the longer form of the constant. This also simplifies modifying the program, should the value of PI change." Or, quoting the other aphorism, what is constant for someone is a variable for someone else. What is run time for LISP (which lacked good compiler at early stages - the famous two horse asses width constraint) can freely be a compile time for other language. Or for a LISP, for that matter, in our time. The compile time expansion allows for better (because faster) error checking.

abecedarius 8y ago

Take a macro like (infix x + vx * dt). It's not equivalent to calling some function with thunked arguments -- it has to analyze the literal text of the arguments to turn this into (+ x (* vx dt)).

jfoutz 8y ago

i'll defer to lispm's expert opinion, but i always thought of macros as _conditional_ evaluation. `if` is a special form because it does not evaluate the branch not taken. it's tricky to implement `if` without using `if` or `cond`. macros make it easy to implement your own special forms. in your example, you might want to implement `first`. (first b c) -> (eval b) i can't think of a way (but it might be possible) to not evaluate c with a regular function. first is trivial and dumb, but it's obvious how to turn that into short circuiting `and` or `or`.

bitwize 8y ago
Macros are more general than that; they are generalized expression rewriters. `if` is tricky to implement in general because you need a primitive to select an alternative based on whether a given value is truthy or falsy. Let's pretend we have such a primitive that's not a special form called `call-if`. Then you can implement `if` in terms of `call-if` by writing it as a macro that transforms this: (if (condition? x) (do-a x) (do-b x)) into this: (call-if (condition? x) (lambda () (do-a x)) (lambda () (do-b x))) (The semantics of `call-if`, by the way, are exactly the semantics exposed by the Smalltalk ifTrue:ifFalse: family of methods.) But macros can do more than merely wrap things in lambdas, which alone is sufficient for delayed and conditional evaluation. They can quote identifiers that are passed into them, allowing you to write new defining forms and even introduce new bindings. A nontrivial, but understandable, example is the `define-record-type` special form from SRFI 9, which can define a new disjoint record type and a constructor, predicate, and field accessors for that type all in one form. If you are willing to break hygiene and have `defmacro` or something like `syntax-case` available, you can even introduce new identifiers from out of nowhere, and write something like Common Lisp's `defstruct` which lets you define a type and automatically derive names for the constructor, slot accessors, etc. from the names of the type and slots. Macros are a tremendously powerful, general code generation and rewriting tool, built right into the language. Underestimate them at your own peril!
- jfoutz 8y ago
  aww, now i'm embarrassed about my lack of imagination. I have to admit i forgot about the magic you can do with define. Excellent point. i kinda sorta think defun in common lisp injects the new function at top level regardless, but in scheme it's scoped. So in CL you could probably get away with a simple lambda (or function) to make your struct syntax, scheme would need the macro to put all of that stuff at the top level. Yeah, macros are crazy powerful

kazinator 8y ago

Macros determine what requirements apply to the processing of syntax. Which parts of that syntax, if any, are evaluated according to what rules, in what environment.

zimablue 8y ago

I think it's "'delayed evaluation' + 'transparent internal structure'". So delayed evaluation isn't enough, because the only thing you can do with N closures is to call them in any order, you can't create a complex new expression dependent on the substructure of the closures. The closest thing to a macro in (eg. python) imo looks like this. Instead of d = c * (a + b) class Adder: def __init__(self, left, right): self.left=left self.right = right class Multiplier: def __init__(self, left, right): self.left = left self.right = right d = Multiplier(Adder(a, b), c) Fill out the __call__ function and/or create an evaluator that recursively calls each class instance with it's parameters and you've made a system that works a lot like a lisp with a macros. Now you can do broadly /anything/, eg. replace every instance of an adder in the call tree with an adder+1. That's using the second required element of my description of macros- the complex substructure, to perform arbitrary transforms. You can't do that with just closures, unless they make their bound parameters accessible (and normally they don't directly, in python they do because it's absurdly dynamic). Of course this is not at compile-time, it's at "delayed execution time", but the difference is kind of slight imo. The benefit of a real lisp over doing this in python is that you don't need to rewrite every fn, operator, control flow statement into your own abstracted level by hand, and it's much faster. Also you can complete the loop and trivially write it back down into eval-able source code text.

kd0amg 8y ago

What are lisp macros for if not to delay evaluation? One class of use cases not covered in the blog post is that a variable appearing as a function's argument can only be a bound occurrence, a use of some name already in scope. A variable appearing as a macro's argument can be the binding occurrence of that name. This is why (using Racket examples) match, define-struct, for/list, and such things aren't functions. [1]: https://docs.racket-lang.org/reference/match.html?q=match#%28form._%28%28lib._racket%2Fmatch..rkt%29._match%29%29 https://docs.racket-lang.org/reference/match.html?q=match#%2... [2]: https://docs.racket-lang.org/reference/define-struct.html?q=define-struct#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._define-struct%29%29 https://docs.racket-lang.org/reference/define-struct.html?q=... [3]: https://docs.racket-lang.org/reference/for.html?q=for%2Flist#%28form._%28%28lib._racket%2Fprivate%2Fbase..rkt%29._for%2Flist%29%29 https://docs.racket-lang.org/reference/for.html?q=for%2Flist...