11 ms·
Lisp Macros, Delayed Evaluation and the Evolution of Smalltalk
- lispm 8y agoThat's a common misconception that Lisp macros are mostly used to 'delay' evaluation. What Smalltalk calls 'blocks' are just (anonymous) functions in Lisp. Books like SICP explain in detail how to use that for delayed evaluation in Lisp/Scheme: https://mitpress.mit.edu/sites/default/files/sicp/full-text/book/book-Z-H-24.html#%_sec_3.5 https://mitpress.mit.edu/sites/default/files/sicp/full-text/...
- i_feel_great 8y agoSmalltalk blocks are anonymous functions sure, but the evaluation is delayed - you have to pass the message #value (and its variants) to it. This not the case in Scheme and maybe Lisps, where evaluation is eager. Passing Smalltalk blocks and Scheme/Lisp lambdas into functions have different behaviours.
- lispm 8y agoThat doesn't matter. One can write IF as a function taking functions as THEN and ELSE forms: CL-USER 9 > (defun my-if (test then else) (if test (funcall then) (funcall else))) MY-IF CL-USER 10 > (let ((a 10)) (my-if (> a 5) (lambda () (print 'hello)) (lambda () (print 'world))) 'done) HELLO DONE The lambdas you see above serve the same purpose as blocks in Smalltalk.
- skybrian 8y agoIt works, but the syntax isn't nice enough to use as-is, so in practice you need a macro. The innovation here is using better syntax for anonymous functions to make such things practical in languages without macros.
- lispm 8y agoIf you think brackets are the core innovation of Smalltalk to make lambdas practical, then Lisp has no problem providing it, too: CL-USER 19 > (defun bracket-reader (stream char) (declare (ignore char)) `(lambda () ,@(read-delimited-list #\] stream t))) BRACKET-READER CL-USER 20 > (set-macro-character #\[ #'bracket-reader) T CL-USER 21 > (let ((a 10)) (my-if (> a 5) [(print 'hello)] [(print 'world)]) 'done) HELLO DONE With a bit more effort, we could also parse a parameter list. But Lisp does not go the route of making IF a function, because it typically provides three different types of language expressions and IF then is a special operator: 1) function calls 2) macro forms 3) a small set of special operators, which are implemented as built-in functionality. One of them is a core conditional operator. More complex conditional operators then are implemented as macros, which expand eventually into the core conditional operator. The interpreter and compiler will have to specially recognize and implement these special operators. See IF: http://clhs.lisp.se/Body/s_if.htm http://clhs.lisp.se/Body/s_if.htm The usual Lisp view is that [(print 'hello)] is no useful improvement over (lambda () (print 'hello)) Some disagree, but most of the time the pattern (<operator> <expr0> ... <exprn>) is preferred in Lisp over <fancy-character> <expr0> ... <exprn> <another-fancy-character>
- skybrian 8y agoOkay, but if you have macros, you might as well use them. The idea is more about exploring what can be done when designing a less powerful language, without macros.
- lispm 8y ago> The idea is more about exploring what can be done when designing a less powerful language, without macros. Lot's of things. But the language won't be able to easily compute code at compile time, which a main purpose of macros. Lisp has a simple data format for source code and thus code transformations are relatively simple to do and even integrated into the language.
- reitzensteinm 8y agoWhat are lisp macros for if not to delay evaluation? I assume you're making a distinction between purpose and mechanism here? I've always understood the semantics to be: fn: (a b c) => (call a (eval b) (eval c)) mac: (a b c) => (eval (call a b c))
- lispm 8y agoThat might be more like what was called an FEXPR mechanism in some early Lisps or some niche Lisps, or even in languages like R. https://en.wikipedia.org/wiki/Fexpr https://en.wikipedia.org/wiki/Fexpr When an FEXPR is called, it gets unevaluated args and can then at runtime decide what to do. Macros OTOH are a different mechanism, where expressions get rewritten at macro expansion time, which can be for example at compile time. The macro then gets called with arbitrary expressions it encloses (those don't need to be valid code by themselves) and computes new source. Thus macros are code generators from expressions. In a compiled implementation, macro expansion is done with compilation interleaved: each form gets expanded, even repeatedly until it no longer expands into a macro, and then the resulting non-macro form gets compiled. Thus in a way a macro does not delay execution, it does the opposite: it actually shifts computation to compile time -> the computation of code from expressions and the computation of arbitrary side effects in the compile-time environment. In an interpreter version of Lisp, the macro gets also expanded at runtime - but in its own macroexpansion during evaluation. There eval will call the macroexpander repeatedly until it gets a non-macro form. Now, what can you do with arbitrary complex code generators at compile time? https://stackoverflow.com/a/2563308/69545 https://stackoverflow.com/a/2563308/69545 Actually Paul Graham wrote a classical Lisp book explaining a bunch of things around macros. Available here for download: http://www.paulgraham.com/onlisp.html http://www.paulgraham.com/onlisp.html The classic Lisp article which motivated the move from FEXPRs to macros: https://www.nhplace.com/kent/Papers/Special-Forms.html https://www.nhplace.com/kent/Papers/Special-Forms.html
- reitzensteinm 8y agoI don't think what I wrote is a FEXPR, which doesn't evaluate the result of the function call. I'm having a hard time parsing what you wrote or any of the sources you linked in such a way that says the semantics of macros differs from what I wrote (certainly the implementation gets a lot more complex and there are subtleties).
- mpweiher 8y ago> misconception that Lisp macros are mostly used to 'delay' evaluation. It was in the examples. No generalization to all of LISP was made or intended, though it would be interesting to make a survey study of actual macro usage by developers. It is also clear that you actually need the delay-evaluation mechanism in order to get something like "if" out of the language and into the library. > what Smalltalk calls 'blocks' are just (anonymous) functions in Lisp "but blocks/anonymous functions seem quite different from alternate argument-passing mechanisms." I think the history also explains why Smalltalk blocks were a bit odd as anonymous functions go: they did not start out as a "function" mechanism, they started out as just a delayed evaluation mechanism, and Smalltalk has no ordinary functions, just the anonymous kind. Which is also odd. The Smalltalk developers were well aware of LISP, cite it as a major source of inspiration in fact. They initially had a more macro-ish mechanism (the unevaluated token-stream), then moved to a pure delayed-evaluation mechanism which then turned into blocks. They never added a macro mechanism again. I think that's interesting. YMMV.
- lispm 8y ago> It is also clear that you actually need the delay-evaluation mechanism in order to get something like "if" out of the language and into the library. In a functional programming language one can just use functions. (defun l-true (a b) a) (defun l-false (a b) b) (defun l-if (f a b) (funcall f a b)) True is a function, false is a function and if is a function. Now we can do conditional execution: CL-USER 57 > (l-if #'l-true 1 2) 1 CL-USER 58 > (l-if #'l-false 1 2) 2 CL-USER 59 > (funcall (l-if #'l-true (lambda () (print 'true-case) nil) (lambda () (print 'false-case) nil))) TRUE-CASE NIL But Lisp doesn't do that. It provides one conditional as a primitive.
- mpweiher 8y ago>> need delay-evaluation [for "if" in lib] > one can just use functions Yes. Obviously. Of course. That's how Smalltalk does it, though it doesn't have general functions, it only has anonymous functions aka (roughly) blocks. Any other language that can pass functions or some sort of equivalent as arguments can obviously also do this. Heck, you can even do this, horribly, in C: #include <stdio.h> void a() { printf("true\n"); } void b() { printf("false\n"); } typedef void (*fnfn) (); void fnif(int cond, fnfn trueCase, fnfn falseCase ) { fnfn switcher[2]={ falseCase, trueCase }; switcher[cond](); } int main(int argc, char *argv[] ) { fnif( 0, a,b ); } (A little surprised that this compiled on the first try. More surprised that clang didn't show any warnings, even with -Wpedantic. Shows you how well I understand C/C compilers ¯\_(ツ)_/¯ ). And function definitions, lambdas, blocks etc. all delay the evaluation of their bodies.
- garganzol 8y agoDid you ever try to implement a macro in a Lisp? If you ever try, you'll quickly find out that a macro is essentially a lambda without argument evaluation. That's it. If we take another look on it, yep, that's a form of delayed evaluation of lambda parameters, however I would prefer the term "delayed expansion".
- phoe-krk 8y agoRelated discussion: https://news.ycombinator.com/item?id=14333824 https://news.ycombinator.com/item?id=14333824
- chriswarbo 8y ago> If you ever try, you'll quickly find out that a macro is essentially a lambda without argument evaluation. That's it. > If we take another look on it, yep, that's a form of delayed evaluation of lambda parameters, however I would prefer the term "delayed expansion". To me, 'delayed evaluation' implies something like call-by-need (e.g. Haskell) or call-by-name (e.g. Algol60), where we can't distinguish an evaluated argument from an unevaluated one, since any attempt to 'look at' (i.e. branch on the value of) the argument will force it to be evaluated. For example, if `(expensive-calculation)` evaluates to `42`, there could be no difference in the behaviour of `(my-delayed-function (expensive-calculation))` and `(my-delayed-function 42)` (although there would presumably be a difference in running time). In contrast, Lisp macros let us inspect the AST without forcing evaluation, so we can distinguish between these two values, and hence `(my-macro (expensive-calculation))` might behave completely differently to `(my-macro 42)`.
- aidenn0 8y agoConsidering the username of the person you are replying to, you might consider that they are very likely to have implemented macros in a lisp. In common lisp, defining a macro defines a function that receives as its arguments the unevaluated forms passed to it plus (optionally) the lexical environment. This part is indeed "essentially a lambda without argument evaluation." The magic isn't there, the magic is in the interpretation of the value it returns. It can return any arbitrary lisp forms. This allows macros to do far more than just delay evaluation. Pretty much anything that involves code walking is not possible with lambdas, for example. Some setf expanders would be impossible with lambdas as well.
- syastrov 8y agoScala has implemented support for async-await syntax using its experimental macros: https://docs.scala-lang.org/sips/async.html https://docs.scala-lang.org/sips/async.html Just an example of how powerful macros can be. They also have an example in their docs about implementing printf using a macro so that formatting parameters are typechecked.
- supernintendo 8y agoMacros work much better in homoiconic languages like Lisp than they do in other languages. Most of the code I write these days is in Elixir and I avoid macros unless absolutely necessary. The benefits generally don't outweigh the (long-term) costs.
- lispm 8y agoOne of the differences between macros in Lisp and some other languages which provide macros is that the expressions themselves don't need to be valid code in some programming language -> they don't get parsed by a language parser upfront. Thus I can write a postfix macro and then write code like this: (postfix 2 3 + 3 *) even though Lisp requires code to be nested prefix expressions. Some other languages won't allow this, because the expression 2 3 + 3 * is not legal in their language. Thus it only may allow macro transformations from legal expressions to other legal expressions...
- mpweiher 8y ago> don't need to be valid code Yes. One of the examples from the talk was a comment macro. Very cool (and the talk was about fun/cool stuff, not about practicalities). The question is whether you want that sort of power in day-to-day programming. My guess is no. That's also what the PARC/LRG folks found out with Smalltalk-72. It's also something I hear from some very seasoned LISP hackers. It's also the sense I am getting from these very powerful DSL/LOP workbenches. From TFA: The reason the question is relevant is, of course, that although it is fun to play around with powerful mechanisms, we should always use the least powerful mechanism that will accomplish our goal, as it will be easier to program with, easier to understand, easier to analyse and build tools for, and easier to maintain. It's also why I use the C pre-processor very reluctantly. Though I do use it. From time to time. And then try to get rid of that use if I can[1] And no need to explain how much better LISP macros are :-) In a sense, like Smalltalk, LISP may be just too powerful, in the words of Alan Kay "Lisps frequently 'eat their children'" so that there's always an answer (use a macro) that will cut off an interesting question. [1] https://blog.metaobject.com/2018/11/refactoring-towards-language.html https://blog.metaobject.com/2018/11/refactoring-towards-lang...
- panzerklein 8y agoThere is whole set of "with-x" macros that aren't about delayed evaluation.
- User23 8y agoWhat's the with-x macro that isn't providing dynamic extent, that is, delaying execution of some cleanup logic?
- bluefox 8y agoConsider for example CL:WITH-SLOTS, or CL-WHO:WITH-HTML-OUTPUT... such macros establish context, and are not really about delaying evaluation. On Lisp has a section about uses of macros, which is not exhaustive, but shows there's more to them than "delayed evaluation".
- User23 8y agoAlgol 60 is wildly underappreciated, and pass by name[1] is a great example. [1] http://www.cs.sfu.ca/~cameron/Teaching/383/PassByName.html http://www.cs.sfu.ca/~cameron/Teaching/383/PassByName.html Edit: An elementary error is to assume that call by name is equivalent to pass by reference. It's not.
- mpweiher 8y ago"Here is a language so far ahead of its time, that it was not only an improvement on its predecessors, but also on nearly all its successors" - Hoare http://www.computernostalgia.net/articles/algol60.htm http://www.computernostalgia.net/articles/algol60.htm
- agumonkey 8y agoAlways surprises me how much algol did embed.
- j-pb 8y agoLambdas are about delayed evaluation. Macros are about disabling evaluation. Lisp code is just an abstract syntax tree notation format (s-exps) that comes with default evaluation semantics. Macros allow you to disable those evaluation semantics to reuse the AST for a different programming language. So for example to add pattern matching facilities to the language, you come up with a syntax and its representation in s-exp AST and then write a macro that describes the unification operation using the default semantics. Same for logic programming or any other paradigm not initially supportet. Languages like clojure also bootstrap quite a bit of the language from a simpler to implement dialect. (look at all the functions with a * in clj source they're the base language)
- lispm 8y agolambdas are anonymous functions. Macros are code transformers. Lisp code is not an AST.
- j-pb 8y agoDifferent name for the same thing, whats your point? Lambdas/AF are used to delay evaluation but keep the default semantics. Macros are more than simple code transformers, that wording somehow implies that they somehow retain the semantics of the data passed to them, which mighy be the case but is not required at all. S-Expressions are just a serialisation format for the m-expression AST.
- lispm 8y agoLambdas are just functions. Delaying functionality is just one use of functions. > that wording somehow implies that they somehow retain the semantics of the data passed to them Since they can do arbitrary transformations, retaining semantics is not in focus. Since the input may not have any semantics defined, the semantics is actually provided via the macro implementation. > S-Expressions are just a serialisation format for the m-expression AST. S-expressions know nothing about 'syntax'. Thus they can't be an 'abstract syntax tree'. (3 4 +) is a valid s-expression, but carries no information about any syntax (what is the + ? in an s-expression it's just a symbol) and is also an invalid Lisp expression. An abstract syntax tree would be the result of parsing a program according to some grammar and it would represent the syntactic categories. The parsing stage would already eliminate invalid programs of that programming language. https://upload.wikimedia.org/wikipedia/commons/thumb/c/c7/Abstract_syntax_tree_for_Euclidean_algorithm.svg/800px-Abstract_syntax_tree_for_Euclidean_algorithm.svg.png https://upload.wikimedia.org/wikipedia/commons/thumb/c/c7/Ab... In Lisp it is important that the s-expression is NOT a syntax tree. Otherwise it would be difficult to write macros which violate Lisp syntax.
- stevelosh 8y agoLisp user here. I'll chime in with another example of using macros for more than just delaying evaluation. I wrote a library called Chancery[1] for procedurally generating strings (and other data). It's inspired by Tracery[2] but takes advantage of macros to make it easier to read and feel more like part of the language. I use it to write stupid Twitter bots like https://twitter.com/git_commands https://twitter.com/git_commands and https://twitter.com/rpg_shopkeeper https://twitter.com/rpg_shopkeeper for fun. As an example let's say we want to generate a message about the loot we receive from a monster in a fantasy, D&D-style story. Maybe we'll start with some random weapons: (chancery:define-string weapon-type "sword" "spear" "lance" "flail" "mace") (weapon-type) ; => "mace" (weapon-type) ; => "sword" This expands like so: (macroexpand-1 '(chancery:define-string weapon-type "sword" "spear" "lance" "flail" "mace")) ; => (DEFUN WEAPON-TYPE () (CASE (CHANCERY::CHANCERY-RANDOM 5) (0 "sword") (1 "spear") (2 "lance") (3 "flail") (4 "mace"))) This simple case actually could be done just by delaying evaluation, as long as we get every body clause as a separate thunk. Now let's define a rule for generating the material of a weapon: (chancery:define-string (weapon-material :distribution :weighted) (100 "iron") (40 "steel") (5 "silver") (4 "gold") (1 "adamantine")) This will generate the materials according to a weighted distribution, and macroexpands to: (DEFUN WEAPON-MATERIAL () (CASE (CHANCERY::WEIGHTLIST-RANDOM #<CHANCERY::WEIGHTLIST ((100 0) (40 1) ...)>) (0 "iron") (1 "steel") (2 "silver") (3 "gold") (4 "adamantine"))) This case needs more than just delayed evaluation. If you receive `(100 "iron")` as an opaque thunk, where all you can do is evaluate it, there's no way to pull out the weight and body components. If we add a few more rules, we can see more cases where we need to go beyond delayed evaluation: (defun currency-amount () (+ 10 (random 100))) (chancery:define-string (currency-type :distribution :zipf) "copper" "silver" "gold" "platinum") (chancery:define-string loot #((weapon-material weapon-type) chancery:a) (currency-amount currency-type "coins")) (chancery:define-string discovery ("You open the chest and find" loot :. ".") ("You find" loot "in the monster's hidden stash.") ("You find nothing but dust and cobwebs.")) (discovery) ; => "You find nothing but dust and cobwebs." (discovery) ; => "You find an iron sword in the monster's hidden stash." (discovery) ; => "You find 61 copper coins in the monster's hidden stash." (discovery) ; => "You open the chest and find a steel sword." Macroexpanding the last one: (DEFUN DISCOVERY () (CASE (CHANCERY::CHANCERY-RANDOM 3) (0 (CHANCERY::JOIN-STRING "You open the chest and find" (PRINC-TO-STRING #\ ) (LOOT) ".")) (1 (CHANCERY::JOIN-STRING "You find" (PRINC-TO-STRING #\ ) (LOOT) (PRINC-TO-STRING #\ ) "in the monster's hidden stash.")) (2 (CHANCERY::JOIN-STRING "You find nothing but dust and cobwebs.")))) Here we can see the macro walking the lists and doing different things to each element: strings are included raw, symbols are turned into function calls, and the special keyword :. suppresses the usual joining space character inserted between everything. There's also some special handling of vectors, in the LOOT example, which I won't go into. This is more than just delayed evaluation — we're inspecting the actual structure of the code received by the macro at macroexpansion time. If all we had were an opaque thunk that we could evaluate later, we couldn't do this. Delayed evaluation is enough for certain kinds of abstraction, like writing basic control structures, but isn't as powerful as full macros. Macros let you transform arbitrary code into other arbitrary code using the full power of the language. [1]: https://sjl.bitbucket.io/chancery/ https://sjl.bitbucket.io/chancery/ [2]: http://tracery.io/ http://tracery.io/