5 ms·

I'm wanting to learn lisp, can someone eli5 metaprogramming?

speed_spread 4y ago
> eli5 metaprogramming Metaprogramming is "programming programming". Consider "programming" as writing a series of steps for a computer to follow. Metaprogramming is writing a series of steps for a computer to follow to write a series of steps for a computer to follow. When working with LISP, metaprogramming is just a normal way to program things that's built into the language. Rather than writing everything as a linear program as you would in conventional languages, you can write a program that "writes itself" (or modifies parts of itself). While you could meta-program in many languages (using code generators and whatnot), in LISP this is easy because the programs themselves are written in a way that makes them easy to be manipulated by other parts of the program. This is why you'll see so many parentheses when looking at LISP code. The simplicity of the syntax is also what makes it so powerful. Metaprogramming is a technique that often results in much shorter programs which can make single programmers very effective. The downside is that the higher level of abstraction (leaving ELI5 zone here, sorry) can make the workings of LISP programs quite opaque to newcomers as things seem a lot like magic. Therefore preserving the metaprogramming advantages in larger teams requires _especially_ good documentation and attention to design.
Shorel 4y ago
You write a program that writes part of the program for you.
timonoko 4y ago
Old lisp makers rule was "macro is like lambda, but with one extra eval in the original environment". In other words: if your function is able to produce (+ 1 2), same function produces 3, when used as macro.
- rurban 4y ago
  macro is not like lambda. macros are like special forms. a lambda evaluates all it's args, a macro not.
  - lispm 4y ago
    fexprs
  - timonoko 4y ago
    My Bad. I have special nlambda, which does not evaluate anything at any stage. It was originally for testing macros without macroexpand, which expanded too much.
    - timonoko 4y ago
      I have also nslambda, which take only one argument. I think the macro in some ancient lisp was also like that. My macroes are called mlambda because of this confusion.
reikonomusha 4y ago
Sometimes you're writing code, and you want to be able to express an idea that your language does not elegantly support. But you know how to express the idea, but it takes a lot of boilerplate, or is just a massive nuisance to write. Metaprogramming allows you to write a function that generates the boilerplate, and transparently use that function as if it's a brand new keyword in your programming language. That function in metaprogramming circles is called a "macro", not to be confused with C macros. Stated another way: a macro = a function that takes as input the code you want to write, and provides as output code that your programming language can actually execute. (It's like a super miniature compiler.) For example, let's pretend C is metaprogrammable. Perhaps I want a new keyword "pfor" which is like "for" but executes the loop in parallel. int x[20]; pfor(i = 0; i < 10; i++) { x[2*i] = expensive(2*i); x[2*i+1] = expensive(2*i+1); } Here, we assume "expensive" is some really expensive function that maybe might take several seconds to execute, which is why we want to parallelize it. This "pfor" is just a sort is pseudo code. In reality, you know how we would actually have to write this: make a loop in which we spawn 10 threads, set i to be a thread local variable in each, call a callback function in each thread void thread_callback(int i){ x[2*i] = expensive(2*i); x[2*i+1] = expensive(2*i+1); } and join the threads at the end. This would have the equivalent effect as our desired "pfor" keyword. In a metaprogramming language, we can add this keyword by just telling the compiler what code to generate any time it sees "pfor". We could write something like macro pfor(var = init, cond, incr, body...) { symbol callback = generate_random_symbol(); ``` void $callback(int $var) { $...body; } vector threads; for($var = $init; $cond; $incr) { threads.push(make_thread($callback, $var)); } for(t in threads) join_thread(t); ``` } The triple ``` starts a template, and the $ escapes a template. We have to generate a fresh symbol for the callback name so that they don't collide when we use "pfor" multiple times. Now every time we use "pfor", the compiler will look at our homemade code generator function and write out the code for us. So in effect by telling the compiler how "pfor" can be implemented in more basic code, we've added "pfor" to the language. This isn't a separate tool however! It's not a pre-processor or anything like that. It's built in to the language (hence why it's a programming language paradigm).
- hutzlibu 4y ago
Jtsummers 4y ago
It can mean a few different things, in this case it's about programs that manipulate the program source and produce new programs (or program segments). Think of a code generator that can take a DSL and turn it into some valid (for execution) code: for i in 1 to 10 do printf("%d\n", i) Becomes (through a preprocessor in this hypothetical system) valid C as: for(int i = 1; i <= 10; i++) { printf("%d\n", i); } Now that's not too interesting, but it is a bit (subjective) nicer looping syntax than C's straight syntax. In the Lisp case, macros allow you to take an arbitrary set of values (the parameters to the macro) and produce some other value or values in their place. This could be just executing the code to get a kind of compile time evaluation: (hypotehtical-expression-evaluator-macro (complex-computation-that-gets-reused-100-times-but-takes-10-minutes-each-time)) Or it could be more transformative, like Lisp's loop macro that lets you write an Algol-ish loop instead of a lisp-y loop (or using recursion or other approaches): (loop for i from 1 to 10 do (print i)) Which gets transformed into some other valid Lisp code (not on my laptop so not in a position to actually show the result). David Moon's "Moon's Mark Down" is actually a pretty neat case of Lisp meta programming. He defines a macro defdot which will take a name for a particular directive for his markdown syntax and generate all the necessary functions so that things "just work". And when parsing, when a . is found at the start of a line, the correct handler is calculated and called. To add a new directive, you add a new defdot and that macro will create the relevant functions and register the new handler. http://users.rcn.com/david-moon/MMD/HTML/index.html http://users.rcn.com/david-moon/MMD/HTML/index.html http://users.rcn.com/david-moon/MMD/MMD.lisp http://users.rcn.com/david-moon/MMD/MMD.lisp

> eli5 metaprogramming Metaprogramming is "programming programming". Consider "programming" as writing a series of steps for a computer to follow. Metaprogramming is writing a series of steps for a computer to follow to write a series of steps for a computer to follow. When working with LISP, metaprogramming is just a normal way to program things that's built into the language. Rather than writing everything as a linear program as you would in conventional languages, you can write a program that "writes itself" (or modifies parts of itself). While you could meta-program in many languages (using code generators and whatnot), in LISP this is easy because the programs themselves are written in a way that makes them easy to be manipulated by other parts of the program. This is why you'll see so many parentheses when looking at LISP code. The simplicity of the syntax is also what makes it so powerful. Metaprogramming is a technique that often results in much shorter programs which can make single programmers very effective. The downside is that the higher level of abstraction (leaving ELI5 zone here, sorry) can make the workings of LISP programs quite opaque to newcomers as things seem a lot like magic. Therefore preserving the metaprogramming advantages in larger teams requires _especially_ good documentation and attention to design.

Shorel 4y ago

You write a program that writes part of the program for you.

timonoko 4y ago

Old lisp makers rule was "macro is like lambda, but with one extra eval in the original environment". In other words: if your function is able to produce (+ 1 2), same function produces 3, when used as macro.

rurban 4y ago
macro is not like lambda. macros are like special forms. a lambda evaluates all it's args, a macro not.
- lispm 4y ago
  fexprs
- timonoko 4y ago
  My Bad. I have special nlambda, which does not evaluate anything at any stage. It was originally for testing macros without macroexpand, which expanded too much.
  - timonoko 4y ago
    I have also nslambda, which take only one argument. I think the macro in some ancient lisp was also like that. My macroes are called mlambda because of this confusion.

reikonomusha 4y ago

Sometimes you're writing code, and you want to be able to express an idea that your language does not elegantly support. But you know how to express the idea, but it takes a lot of boilerplate, or is just a massive nuisance to write. Metaprogramming allows you to write a function that generates the boilerplate, and transparently use that function as if it's a brand new keyword in your programming language. That function in metaprogramming circles is called a "macro", not to be confused with C macros. Stated another way: a macro = a function that takes as input the code you want to write, and provides as output code that your programming language can actually execute. (It's like a super miniature compiler.) For example, let's pretend C is metaprogrammable. Perhaps I want a new keyword "pfor" which is like "for" but executes the loop in parallel. int x[20]; pfor(i = 0; i < 10; i++) { x[2*i] = expensive(2*i); x[2*i+1] = expensive(2*i+1); } Here, we assume "expensive" is some really expensive function that maybe might take several seconds to execute, which is why we want to parallelize it. This "pfor" is just a sort is pseudo code. In reality, you know how we would actually have to write this: make a loop in which we spawn 10 threads, set i to be a thread local variable in each, call a callback function in each thread void thread_callback(int i){ x[2*i] = expensive(2*i); x[2*i+1] = expensive(2*i+1); } and join the threads at the end. This would have the equivalent effect as our desired "pfor" keyword. In a metaprogramming language, we can add this keyword by just telling the compiler what code to generate any time it sees "pfor". We could write something like macro pfor(var = init, cond, incr, body...) { symbol callback = generate_random_symbol(); ``` void $callback(int $var) { $...body; } vector threads; for($var = $init; $cond; $incr) { threads.push(make_thread($callback, $var)); } for(t in threads) join_thread(t); ``` } The triple ``` starts a template, and the $ escapes a template. We have to generate a fresh symbol for the callback name so that they don't collide when we use "pfor" multiple times. Now every time we use "pfor", the compiler will look at our homemade code generator function and write out the code for us. So in effect by telling the compiler how "pfor" can be implemented in more basic code, we've added "pfor" to the language. This isn't a separate tool however! It's not a pre-processor or anything like that. It's built in to the language (hence why it's a programming language paradigm).

hutzlibu 4y ago

Jtsummers 4y ago

It can mean a few different things, in this case it's about programs that manipulate the program source and produce new programs (or program segments). Think of a code generator that can take a DSL and turn it into some valid (for execution) code: for i in 1 to 10 do printf("%d\n", i) Becomes (through a preprocessor in this hypothetical system) valid C as: for(int i = 1; i <= 10; i++) { printf("%d\n", i); } Now that's not too interesting, but it is a bit (subjective) nicer looping syntax than C's straight syntax. In the Lisp case, macros allow you to take an arbitrary set of values (the parameters to the macro) and produce some other value or values in their place. This could be just executing the code to get a kind of compile time evaluation: (hypotehtical-expression-evaluator-macro (complex-computation-that-gets-reused-100-times-but-takes-10-minutes-each-time)) Or it could be more transformative, like Lisp's loop macro that lets you write an Algol-ish loop instead of a lisp-y loop (or using recursion or other approaches): (loop for i from 1 to 10 do (print i)) Which gets transformed into some other valid Lisp code (not on my laptop so not in a position to actually show the result). David Moon's "Moon's Mark Down" is actually a pretty neat case of Lisp meta programming. He defines a macro defdot which will take a name for a particular directive for his markdown syntax and generate all the necessary functions so that things "just work". And when parsing, when a . is found at the start of a line, the correct handler is calculated and called. To add a new directive, you add a new defdot and that macro will create the relevant functions and register the new handler. http://users.rcn.com/david-moon/MMD/HTML/index.html http://users.rcn.com/david-moon/MMD/HTML/index.html http://users.rcn.com/david-moon/MMD/MMD.lisp http://users.rcn.com/david-moon/MMD/MMD.lisp