5 ms·

As unsatisfying as it may sound, Common Lisp taught me that, truly, none of this function binding capture stuff really matters in practice. Millions of lines of

As unsatisfying as it may sound, Common Lisp taught me that, truly, none of this function binding capture stuff really matters in practice. Millions of lines of Common Lisp code have been running for decades without running into problems with capture. So I purport solving this problem is akin to solving a 0.00000001% issue if we measure the frequency of encountering this error writing thousands of lines of Lisp per day for a sizable chunk of one’s lifetime. The explanations by Pitman et al. are wise. If someone on my team were regularly writing macros that expanded into non-externalizable, non-PRINTable function objects, I’d probably be irritated. You’re hindering my pretty-printer, you’re hosing my debugging tools, and you’re breaking my expectations of interactive development. What? One big negative the article doesn’t cover is that if you’re essentially embedding function pointers in your macros, redefining that function won’t take effect in previously expanded code. (defun f () 1) (defmacro m () `(f)) (defmacro n () `(,#'f)) (defun g () (m)) (defun h () (n)) ; later (defun f () 2) H will always return 1 (bad!). G will reflect the global binding (good!). You would have to re-evaluate the source definition of H to get it to reflect the new pointer. This is contrary to Lisp’s DNA of interactive development and is a big no-no. (You have similar issues if you, a library author, are declaring the functions you provide to application programmers as globally INLINE. It’s not for you, the library author, to choose when things should be inlined!) You can force late binding using FUNCALL on a symbol to look up the global binding (defmacro n++ () `(funcall 'f)) if you so please, but it’s not very satisfying. F can never be captured by the local environment there.

lukego 5y ago
Similar to variable capture with dynamic scope e.g. in classical Emacs Lisp? That's been the root cause of a bug for me exactly once in years of Emacs hacking, and having finally been bitten my radar detects that risk going forward. So much as I prefer lexical scope the alternative seems pretty tame in practice.
- TeMPOraL 5y ago
  I miss dynamic scope in non-Lisp languages. Perhaps defaulting to it for general-purpose programming was a bad idea (one on which most Lisps backtracked), but it's arguably a good choice for an extensible application like Emacs, and it's definitely a tool you want to have available. An easy way to see this is: dynamic binding is to environmental variables what lexical binding is to command line arguments.
  - lukego 5y ago
    Buffer-local variables are also underrated. There's another universe where this context-switching between objects is what we call "OO" instead of Smalltalk/Self style.
waterhouse 5y ago
> redefining that function won’t take effect in previously expanded code Yeah, that is an important issue. What do you think of the following solution? 1. The semantics of the language are that, every time a function is called, any macros in its body get reexpanded. 2. The language implementation is expected to notice that, in normal cases, neither the macro nor the functions it calls have been redefined, and therefore the previously-expanded (and -compiled) version can be used; but then to notice when that changes (i.e. when you do redefine something) and generate new code accordingly. The feasibility of 2 I'm not sure about. Edit: Regarding the printing of functions, I don't think it would be that difficult to make the pretty-printer, upon encountering a function object, detect whether that function is bound to a global variable, and if so, print the name. (Possibly the name with a "f:" prefix, or in a different color.)
- fiddlerwoaroof 5y ago
  The data needed for (2) is mostly already available: most lisps maintain a Xref database and Slime already has a binding for `slime-who-macroexpands`[1]. Integrating this performantly into the compiler might be interesting, but I think it's theoretically tractable. [1]: https://common-lisp.net/project/slime/doc/html/Cross_002dreference.html https://common-lisp.net/project/slime/doc/html/Cross_002dref...
- reikonomusha 5y ago
  By “print”, I mean it in the Lisp sense of “print readably”. In Lisp, PRINT really means “serialize as text that can be deserialized with READ.” In Common Lisp (and Janet), functions print fine in the colloquial sense: > (print #'print) #<FUNCTION PRINT> The character sequence ‘#<’ means it’s “unreadable” though.
- TeMPOraL 5y ago
  1. might fly in some purely-functional Lisp, but most (all?) real-world Lisps can have side effects in macro expansion. And by "side effects", I don't mean altering some global variable or adding a function to the symbol table - Lisp macros have access to the full language runtime, so a macroexpansion may just as well send some HTTP requests or hit a database. Ironically, the author of the article was correct in their view that he initially described as wrong. Macros are just functions, typically taking code and outputting code. They just didn't understand the evaluation rules surrounding macro invocation.
ianthehenry 5y ago
> I purport solving this problem is akin to solving a 0.00000001% issue It seems like Common Lisp has done quite a lot to solve this problem already, which is why it is a non-issue in Common Lisp. > non-externalizable, non-PRINTable function objects Yes, this would be a very annoying technique in a language with non-externalizable, non-PRINTable functions. > This is contrary to Lisp’s DNA of interactive development and is a big no-no. Redefining functions interactively is actually a great motivating case for cross-stage persistence. Instead of interpolating the value of the function, interpolate the lexical binding of the function itself, and then redefinitions will be reflected automatically in any expansion of the macro.
- reikonomusha 5y ago
  It’s ambiguous to me whether you’re suggesting there are compiled languages with printable and externalizable functions. Given that many useful functions are actually closures over lexical environments possibly shared by other functions, and compiled to native code, I’d be impressed to find a language where such objects are printable and externalizable. I don’t think they are in Janet.
  - waterhouse 5y ago
    > closures over lexical environments possibly shared by other functions The same problem exists with lists (or any other mutable object). If x and y point to the same list, and you print x and read it back in, and then modify the list x points to, then (naively) it won't modify y. If you did want to preserve such structure sharing, one approach would be to print the entire environment and make liberal use of #n= notation. (I've occasionally done things like this.) > compiled to native code Assuming compilation is a deterministic function of the source code and the environment, and assuming the user hasn't changed the environment, it seems printing the source code should suffice, and the runtime can redo the compilation when necessary.
    - reikonomusha 5y ago
      Functions with compiled code referring to address offsets in the closure environment are not printable and I’d hazard to say they can’t be without compromising something else. The function being purportedly serialized is already in a representation very far removed from its source code, and has mutable state that isn’t just from its closure environment. LOAD-TIME-VALUE (in Common Lisp) is another problem that allocates something akin to a C ‘static’ variable. Compiler macros (in Common Lisp) also make things hairier. Lack of run-time representation of reader macros (in Common Lisp) are the cherry on top—though this is only an issue if you want your serialized representation to mimic the source code closely. Lists do have issues of sharing data with other lists, but most times the lists that are sharing data are somehow adjacent to one another allowing the printer to collect shared references. If we allow ourselves the freedom to ignore shared closure environments, L-T-V and other Common Lisp complications, then serializing a function would still be this very expensive recursive procedure to serialize all transitive callees to capture any possible shared state, and I think that is too much to ask for, especially in the context we are speaking (making functions appropriate objects to be printed as a part of an S-expression). It’s all doable if everything is slow, interpreted, and first-class though. Or maybe you’re serializing FORTH words. :)
fiddlerwoaroof 5y ago
I strongly agree here: my experience is that a Lisp-2 with namespaced symbols has very few issues with function-name capture. While, at one point, Lisp-2s may have been for performance or other implementation details, I find that Lisp-2s are much nicer to code in because you just don't worry about name collisions (as long as you know a handful of relatively simple rules about macros).
- kazinator 5y ago
  My experience that a certain systems programming language of Unix origins featuring 1 namespace and an unhygienic token-based macro system also has few issues with function-name capture, even in code bases with seven digit LOCs. That makes me severely disinterested and skeptical about hygienic macros. They are too weird for the little benefit they provide. You can't look at a piece of code and know what it expands to. Hygienic macros break lexical scope, because code produced by a macro being invoked in some file "A.lisp" is seeing lexical variables defined in some different file "B.lisp". Lexical scope must be physically enclosed and contained. If you expand it here, it sees the variables that are here, and not some other ones elsewhere that you don't see here. Given a nesting like (x (y (z a))), if y binds a, that must be like a brick wall; there is no way that the a reference in z can get around it, to connect with a binding of a produced by x. Either that a refers to z's own binding, or else to y. Yet hygienic macros can perpetrate a lexical-scope-destroying wormhole which can do that: the inner a reference can bypass a definition set up by y, and go to the one in x. It's a downright security issue, like going around your company's firewall, or escaping a container or chroot jail or whatever.
  - fiddlerwoaroof 5y ago
    All I know is that in Python, JS and Clojure is I’ve accidentally written code like: def foo(str): v = str(1) . . . Which resulted in head-scratching errors like “str is not a function”. EDIT: I agree that hygienic macros aren't the right way to solve this issue.
    - louai 5y ago

lukego 5y ago

Similar to variable capture with dynamic scope e.g. in classical Emacs Lisp? That's been the root cause of a bug for me exactly once in years of Emacs hacking, and having finally been bitten my radar detects that risk going forward. So much as I prefer lexical scope the alternative seems pretty tame in practice.

TeMPOraL 5y ago
I miss dynamic scope in non-Lisp languages. Perhaps defaulting to it for general-purpose programming was a bad idea (one on which most Lisps backtracked), but it's arguably a good choice for an extensible application like Emacs, and it's definitely a tool you want to have available. An easy way to see this is: dynamic binding is to environmental variables what lexical binding is to command line arguments.
- lukego 5y ago
  Buffer-local variables are also underrated. There's another universe where this context-switching between objects is what we call "OO" instead of Smalltalk/Self style.

waterhouse 5y ago

> redefining that function won’t take effect in previously expanded code Yeah, that is an important issue. What do you think of the following solution? 1. The semantics of the language are that, every time a function is called, any macros in its body get reexpanded. 2. The language implementation is expected to notice that, in normal cases, neither the macro nor the functions it calls have been redefined, and therefore the previously-expanded (and -compiled) version can be used; but then to notice when that changes (i.e. when you do redefine something) and generate new code accordingly. The feasibility of 2 I'm not sure about. Edit: Regarding the printing of functions, I don't think it would be that difficult to make the pretty-printer, upon encountering a function object, detect whether that function is bound to a global variable, and if so, print the name. (Possibly the name with a "f:" prefix, or in a different color.)

fiddlerwoaroof 5y ago
The data needed for (2) is mostly already available: most lisps maintain a Xref database and Slime already has a binding for `slime-who-macroexpands`[1]. Integrating this performantly into the compiler might be interesting, but I think it's theoretically tractable. [1]: https://common-lisp.net/project/slime/doc/html/Cross_002dreference.html https://common-lisp.net/project/slime/doc/html/Cross_002dref...
reikonomusha 5y ago
By “print”, I mean it in the Lisp sense of “print readably”. In Lisp, PRINT really means “serialize as text that can be deserialized with READ.” In Common Lisp (and Janet), functions print fine in the colloquial sense: > (print #'print) #<FUNCTION PRINT> The character sequence ‘#<’ means it’s “unreadable” though.
TeMPOraL 5y ago
1. might fly in some purely-functional Lisp, but most (all?) real-world Lisps can have side effects in macro expansion. And by "side effects", I don't mean altering some global variable or adding a function to the symbol table - Lisp macros have access to the full language runtime, so a macroexpansion may just as well send some HTTP requests or hit a database. Ironically, the author of the article was correct in their view that he initially described as wrong. Macros are just functions, typically taking code and outputting code. They just didn't understand the evaluation rules surrounding macro invocation.

ianthehenry 5y ago

> I purport solving this problem is akin to solving a 0.00000001% issue It seems like Common Lisp has done quite a lot to solve this problem already, which is why it is a non-issue in Common Lisp. > non-externalizable, non-PRINTable function objects Yes, this would be a very annoying technique in a language with non-externalizable, non-PRINTable functions. > This is contrary to Lisp’s DNA of interactive development and is a big no-no. Redefining functions interactively is actually a great motivating case for cross-stage persistence. Instead of interpolating the value of the function, interpolate the lexical binding of the function itself, and then redefinitions will be reflected automatically in any expansion of the macro.

reikonomusha 5y ago
It’s ambiguous to me whether you’re suggesting there are compiled languages with printable and externalizable functions. Given that many useful functions are actually closures over lexical environments possibly shared by other functions, and compiled to native code, I’d be impressed to find a language where such objects are printable and externalizable. I don’t think they are in Janet.
- waterhouse 5y ago
  > closures over lexical environments possibly shared by other functions The same problem exists with lists (or any other mutable object). If x and y point to the same list, and you print x and read it back in, and then modify the list x points to, then (naively) it won't modify y. If you did want to preserve such structure sharing, one approach would be to print the entire environment and make liberal use of #n= notation. (I've occasionally done things like this.) > compiled to native code Assuming compilation is a deterministic function of the source code and the environment, and assuming the user hasn't changed the environment, it seems printing the source code should suffice, and the runtime can redo the compilation when necessary.
  - reikonomusha 5y ago
    Functions with compiled code referring to address offsets in the closure environment are not printable and I’d hazard to say they can’t be without compromising something else. The function being purportedly serialized is already in a representation very far removed from its source code, and has mutable state that isn’t just from its closure environment. LOAD-TIME-VALUE (in Common Lisp) is another problem that allocates something akin to a C ‘static’ variable. Compiler macros (in Common Lisp) also make things hairier. Lack of run-time representation of reader macros (in Common Lisp) are the cherry on top—though this is only an issue if you want your serialized representation to mimic the source code closely. Lists do have issues of sharing data with other lists, but most times the lists that are sharing data are somehow adjacent to one another allowing the printer to collect shared references. If we allow ourselves the freedom to ignore shared closure environments, L-T-V and other Common Lisp complications, then serializing a function would still be this very expensive recursive procedure to serialize all transitive callees to capture any possible shared state, and I think that is too much to ask for, especially in the context we are speaking (making functions appropriate objects to be printed as a part of an S-expression). It’s all doable if everything is slow, interpreted, and first-class though. Or maybe you’re serializing FORTH words. :)

fiddlerwoaroof 5y ago

I strongly agree here: my experience is that a Lisp-2 with namespaced symbols has very few issues with function-name capture. While, at one point, Lisp-2s may have been for performance or other implementation details, I find that Lisp-2s are much nicer to code in because you just don't worry about name collisions (as long as you know a handful of relatively simple rules about macros).

kazinator 5y ago
My experience that a certain systems programming language of Unix origins featuring 1 namespace and an unhygienic token-based macro system also has few issues with function-name capture, even in code bases with seven digit LOCs. That makes me severely disinterested and skeptical about hygienic macros. They are too weird for the little benefit they provide. You can't look at a piece of code and know what it expands to. Hygienic macros break lexical scope, because code produced by a macro being invoked in some file "A.lisp" is seeing lexical variables defined in some different file "B.lisp". Lexical scope must be physically enclosed and contained. If you expand it here, it sees the variables that are here, and not some other ones elsewhere that you don't see here. Given a nesting like (x (y (z a))), if y binds a, that must be like a brick wall; there is no way that the a reference in z can get around it, to connect with a binding of a produced by x. Either that a refers to z's own binding, or else to y. Yet hygienic macros can perpetrate a lexical-scope-destroying wormhole which can do that: the inner a reference can bypass a definition set up by y, and go to the one in x. It's a downright security issue, like going around your company's firewall, or escaping a container or chroot jail or whatever.
- fiddlerwoaroof 5y ago
  All I know is that in Python, JS and Clojure is I’ve accidentally written code like: def foo(str): v = str(1) . . . Which resulted in head-scratching errors like “str is not a function”. EDIT: I agree that hygienic macros aren't the right way to solve this issue.
  - louai 5y ago