3 ms·

Without Lisp-like macros, you need preprocessors or code generators. Something like Yacc/Bison is easily done with suitable macros. In Common Lisp, macros als

Without Lisp-like macros, you need preprocessors or code generators. Something like Yacc/Bison is easily done with suitable macros. In Common Lisp, macros also enable a kind of Aspect Oriented Programming. That's because one can dynamically modify macro expansion using the macroexpand hook. It's a way to modify a code base without changing the source files, which has all sorts of uses. http://clhs.lisp.se/Body/v_mexp_h.htm#STmacroexpand-hookST http://clhs.lisp.se/Body/v_mexp_h.htm#STmacroexpand-hookST One can implement things natively in Common Lisp, like code coverage tools, that require separate tooling in other languages.

kazinator 2y ago
> one can dynamically modify macro expansion using the macroexpand hook That's an incredibly bad idea, and it's not dynamic except in purely interpreted Lisps that expand the same piece of code each time it is executed. The hooks is absolutely global; whatever you put in there must not break anything. From the spec itself: "For this reason, it is frequently best to confine its uses to debugging situations."
- pfdietz 2y ago
  One would bind the macroexpand hook variable around the call to COMPILE or COMPILE-FILE for that code. The binding goes away after that. It wouldn't be global at all.
  - froh 2y ago
    yah dynamic binding id another thing I dearly miss. thread local is like global in that it doesn't support for partial stacks of coroutines or generators. I'd love to have some dynamic _call stack based_ context lookup in languages like Python or c++.
    - gpderetta 2y ago
      With RAII, it is not terribly hard to build dynamic scoping on top of thread_local variables in C++. template<class T> struct scoped_replace { scoped_replace(T& ref, T new_val) : ref(ref), old_val(ref) { ref = std::move(new_val); } ~scoped_replace() { ref = std::move(old_val); } T& ref; T old_val; }; thread_local int val = 0; int foo() { return val; } int bar() { scoped_replace _(val, 42); return foo(); } int main() { foo(); // returns 0 bar(); // returns 42; foo(); // still returns 0 }
      - kazinator 2y ago
        That's a "shallow binding" implementation of dynamic scope which works fine enough; but if you have coroutines and continuations and such, it might not play along. Whichever thread invokes a continuation will see its own current value of the variable. Under "deep binding", where we have a dynamic environment stack, we can save that stack as part of the context of the coroutine, continuation or whatever else. It requires only one word: a pointer to the top of the dynamic environment. Whenever we restore that pointer value, we have the original environment.
jstanley 2y ago
> It's a way to modify a code base without changing the source files Is another way of saying your source files might not do what they say! You're not selling it to me.
- pfdietz 2y ago
  A use case is code coverage: you can instrument your code by compiling it in a dynamic environment where the hook causes extra coverage recording code to be inserted. You'd like to be able to do this without changing the source code itself.
- diffxx 2y ago
  https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...
- mst 2y ago
  Wrapping the existing code in hooks of some sort to enable injecting profiling or whatever is really quite nice. A lot of javascript hot reloading implementations rely on using Babel to do that as part of a build step so they output the wrapped code and your app loads that instead (and it's not that dissimilar to compilers having a debug mode where they output very different code). I would certainly side-eye you for using the relevant common lisp hooks in production without a very convincing reason, but for many development time tasks it's basically the civilised version of widely used techniques.
- praptak 2y ago
  > your source files might not do what they say! The disconnect between what your source file "does" and what it "says" is present in everything above assembly. Change implementation of `foo(...)` and suddenly dozens of source files (none of which is modified!) which make calls like `foo(x)`, `foo(y)` and `foo(y+1)` change what they "do", all this without changing a single line in these files.
  - jerf 2y ago
    There is still a qualitative difference between reading x.y = foo(z, t) reading as "there is a variable labelled x, with a defined type within this scope, that contains a struct member y with a statically-computable offset within x's known memory layout, that will receive a copy of the function foo (which is known at compile time), which is the result of calling foo with a copy of z and t, which are also values with compile-time known types and layouts", even if you as a human may have to look up foo to know what is going on, and the equivalent in Python, which I will break out into a separate paragraph because this is going to get long even though it will still be incomplete: "there is a reference x to an unknown type, which we will call one of several methods to resolve a field called y, where each of these methods may have been dynamically modified by previous code to not match their source implementation, which may either be a concrete value or a "descriptor", which will will have some method call performed on it (with full OO resolution rules on the method call, including the ability for the class structure to have been dynamically modified by code at run time and also all methods in the class structure to have been dynamically modified, and also the method can do anything a method can do including its own further dynamic modifications to arbitrary other structures, and also the result of running class and method decorators on everything in this hierarchy) with the result of calling foo, a values of unknown type that may be a function, method, or arbitrary values with a __call__ value (which itself may perform arbitrary dynamic operations in the process of returning a callable value), with copies of a reference to the values z and t of unknown type and capability, which will be filled in to foo's argument list in additional possibly complicated manners based on foo's definition and whether it uses args or kwargs, and also all of this code including the very statement under question may not exist in source at all because it may have been 100% dynamically constructed at run time itself." And I probably missed some things in the Python too, but even so, Python already has numerous "and actually literally arbitrary amounts of code could be running here", and can readily introduce more. All languages have the issues where you might have to resolve a function call as a human to know what is is doing, but in some languages, you're nearly done after that (C, Go) and in some languages your journey is just beginning (C++, Python).
jjtheblunt 2y ago
> Something like Yacc/Bison is easily done with suitable macros. Is this true? I may have misunderstood what macros can do in terms of remembering state, while applying. Looking it up again to see what i overlooked...
- pfdietz 2y ago
  You can write a macro that expands a grammar definition into the appropriate code. Anything a preprocessor can do, a macro can do like that.
  - jjtheblunt 2y ago
    But it’s the reverse direction that Bison/Yacc does, as it writes code to recognize essentially expanded grammar rules/macros, collapsing existing code into an expandable grammar rule/macro that would correspond. Thinking…if there’s an obvious equivalence
    - pfdietz 2y ago
      Um, what? Bison/yacc generate compilable C from a grammar file. A CL macro would expand to Common Lisp starting from a grammar description. It's not the reverse at all.
dfox 2y ago
The common definition of “Aspect oriented programming” is more or less “just bolt whatever CLOS method combinators already can do onto another language”. That is powerful concept, but maybe somewhat orthogonal to macros, unless you want to do that with macros, which you certainly can.
- pfdietz 2y ago
  That's certainly a form of AOP, but the hook mechanism here would enable arbitrary expansion time code transformations, not just injections at method calls.
EdwardCoffin 2y ago
Gregor Kiczales, the guy behind Aspect Oriented Programming, said something like Common Lisp couldn't attain the level of (aspect oriented) tooling that Java could, because the tools in Common Lisp could not be sure what the Lisp code they were analyzing actually does, but in Java the language is constrained enough that the tools can figure it out. He said this in a MOP retrospective he gave at the Vancouver Lisp User's Group meeting on 28 June 2006 [1]. There's a link to an .ogg file in the link below, and he talks about this at around the 45 minute mark. [1] https://web.archive.org/web/20060821003818/http://bc.tech.coop/blog/060709.html https://web.archive.org/web/20060821003818/http://bc.tech.co...
- kazinator 2y ago
  > but in Java the language is constrained enough that the tools can figure it out So, ironically, the AOP tooling both takes advantage of this issue and fixes it in the same stroke. Once you have an AOP-endowed language like AspectJ, tooling can no longer know what code does

kazinator 2y ago

> one can dynamically modify macro expansion using the macroexpand hook That's an incredibly bad idea, and it's not dynamic except in purely interpreted Lisps that expand the same piece of code each time it is executed. The hooks is absolutely global; whatever you put in there must not break anything. From the spec itself: "For this reason, it is frequently best to confine its uses to debugging situations."

pfdietz 2y ago
One would bind the macroexpand hook variable around the call to COMPILE or COMPILE-FILE for that code. The binding goes away after that. It wouldn't be global at all.
- froh 2y ago
  yah dynamic binding id another thing I dearly miss. thread local is like global in that it doesn't support for partial stacks of coroutines or generators. I'd love to have some dynamic _call stack based_ context lookup in languages like Python or c++.
  - gpderetta 2y ago
    With RAII, it is not terribly hard to build dynamic scoping on top of thread_local variables in C++. template<class T> struct scoped_replace { scoped_replace(T& ref, T new_val) : ref(ref), old_val(ref) { ref = std::move(new_val); } ~scoped_replace() { ref = std::move(old_val); } T& ref; T old_val; }; thread_local int val = 0; int foo() { return val; } int bar() { scoped_replace _(val, 42); return foo(); } int main() { foo(); // returns 0 bar(); // returns 42; foo(); // still returns 0 }
    - kazinator 2y ago
      That's a "shallow binding" implementation of dynamic scope which works fine enough; but if you have coroutines and continuations and such, it might not play along. Whichever thread invokes a continuation will see its own current value of the variable. Under "deep binding", where we have a dynamic environment stack, we can save that stack as part of the context of the coroutine, continuation or whatever else. It requires only one word: a pointer to the top of the dynamic environment. Whenever we restore that pointer value, we have the original environment.

jstanley 2y ago

> It's a way to modify a code base without changing the source files Is another way of saying your source files might not do what they say! You're not selling it to me.

pfdietz 2y ago
A use case is code coverage: you can instrument your code by compiling it in a dynamic environment where the hook causes extra coverage recording code to be inserted. You'd like to be able to do this without changing the source code itself.
diffxx 2y ago
https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...
mst 2y ago
Wrapping the existing code in hooks of some sort to enable injecting profiling or whatever is really quite nice. A lot of javascript hot reloading implementations rely on using Babel to do that as part of a build step so they output the wrapped code and your app loads that instead (and it's not that dissimilar to compilers having a debug mode where they output very different code). I would certainly side-eye you for using the relevant common lisp hooks in production without a very convincing reason, but for many development time tasks it's basically the civilised version of widely used techniques.
praptak 2y ago
> your source files might not do what they say! The disconnect between what your source file "does" and what it "says" is present in everything above assembly. Change implementation of `foo(...)` and suddenly dozens of source files (none of which is modified!) which make calls like `foo(x)`, `foo(y)` and `foo(y+1)` change what they "do", all this without changing a single line in these files.
- jerf 2y ago
  There is still a qualitative difference between reading x.y = foo(z, t) reading as "there is a variable labelled x, with a defined type within this scope, that contains a struct member y with a statically-computable offset within x's known memory layout, that will receive a copy of the function foo (which is known at compile time), which is the result of calling foo with a copy of z and t, which are also values with compile-time known types and layouts", even if you as a human may have to look up foo to know what is going on, and the equivalent in Python, which I will break out into a separate paragraph because this is going to get long even though it will still be incomplete: "there is a reference x to an unknown type, which we will call one of several methods to resolve a field called y, where each of these methods may have been dynamically modified by previous code to not match their source implementation, which may either be a concrete value or a "descriptor", which will will have some method call performed on it (with full OO resolution rules on the method call, including the ability for the class structure to have been dynamically modified by code at run time and also all methods in the class structure to have been dynamically modified, and also the method can do anything a method can do including its own further dynamic modifications to arbitrary other structures, and also the result of running class and method decorators on everything in this hierarchy) with the result of calling foo, a values of unknown type that may be a function, method, or arbitrary values with a __call__ value (which itself may perform arbitrary dynamic operations in the process of returning a callable value), with copies of a reference to the values z and t of unknown type and capability, which will be filled in to foo's argument list in additional possibly complicated manners based on foo's definition and whether it uses args or kwargs, and also all of this code including the very statement under question may not exist in source at all because it may have been 100% dynamically constructed at run time itself." And I probably missed some things in the Python too, but even so, Python already has numerous "and actually literally arbitrary amounts of code could be running here", and can readily introduce more. All languages have the issues where you might have to resolve a function call as a human to know what is is doing, but in some languages, you're nearly done after that (C, Go) and in some languages your journey is just beginning (C++, Python).

jjtheblunt 2y ago

> Something like Yacc/Bison is easily done with suitable macros. Is this true? I may have misunderstood what macros can do in terms of remembering state, while applying. Looking it up again to see what i overlooked...

pfdietz 2y ago
You can write a macro that expands a grammar definition into the appropriate code. Anything a preprocessor can do, a macro can do like that.
- jjtheblunt 2y ago
  But it’s the reverse direction that Bison/Yacc does, as it writes code to recognize essentially expanded grammar rules/macros, collapsing existing code into an expandable grammar rule/macro that would correspond. Thinking…if there’s an obvious equivalence
  - pfdietz 2y ago
    Um, what? Bison/yacc generate compilable C from a grammar file. A CL macro would expand to Common Lisp starting from a grammar description. It's not the reverse at all.

dfox 2y ago

The common definition of “Aspect oriented programming” is more or less “just bolt whatever CLOS method combinators already can do onto another language”. That is powerful concept, but maybe somewhat orthogonal to macros, unless you want to do that with macros, which you certainly can.

pfdietz 2y ago
That's certainly a form of AOP, but the hook mechanism here would enable arbitrary expansion time code transformations, not just injections at method calls.

EdwardCoffin 2y ago

Gregor Kiczales, the guy behind Aspect Oriented Programming, said something like Common Lisp couldn't attain the level of (aspect oriented) tooling that Java could, because the tools in Common Lisp could not be sure what the Lisp code they were analyzing actually does, but in Java the language is constrained enough that the tools can figure it out. He said this in a MOP retrospective he gave at the Vancouver Lisp User's Group meeting on 28 June 2006 [1]. There's a link to an .ogg file in the link below, and he talks about this at around the 45 minute mark. [1] https://web.archive.org/web/20060821003818/http://bc.tech.coop/blog/060709.html https://web.archive.org/web/20060821003818/http://bc.tech.co...

kazinator 2y ago
> but in Java the language is constrained enough that the tools can figure it out So, ironically, the AOP tooling both takes advantage of this issue and fixes it in the same stroke. Once you have an AOP-endowed language like AspectJ, tooling can no longer know what code does