6 ms·

Could you elaborate on how hygienic macros are a minor disaster?

JonChesterfield 3y ago
Macros achieve a limited version of a fexpr with much more complicated semantics. Hygienic macros take that a step further - you get implicit renaming of variables which is generally convenient - in exchange for even more complicated semantics. The abstraction leaks as well. `(reduce and (list #t #t #f))` is probably an error. Calling a macro and calling a function looks the same and sometimes is the same and sometimes falls over. Macros are sacrosanct to lisp. Hygienic macros are a crown jewel of scheme. They're still the wrong thing though. In the beginning there was dynamic scope, possibly by accident. There were also fexpr - pass the unevaluated data and the environment, instead of passing evaluated data, which were also dynamically scoped. Sometimes called nlambda. That was difficult to program with and very difficult to compile. Macros are easier to use than dynamically scoped fexpr. They're much easier to compile and lisp was getting a bit of a kicking for being too slow. There were some papers written, macros were the better choice, and here we are. Lexical scoping turned up a little while after macros won the fight. Lexically scoped fexpr with first class environments are the right thing. Simpler and more capable than macros. First class environments being another thing sacrificed on the alter of performance decades ago. Shutt noticed this and wrote about it at length. It's slightly subtle that a non-hygienic macro is a subset of fexpr. Force inline it at the caller, refuse to run various calls (e.g. pass it to functions), wrap the return value in an eval in the caller environment. A hygienic macro has to do reflective symbol renaming stuff which is also expressible as a fexpr by implementing the symbol renaming rules of scheme, which thus far I don't have the patience for. Qualified as minor, in that the wrong thing we have is still very useful. Disaster in the sense that we could have fexpr if history had happened in a slightly different sequence. Opportunity in that lisp can be better. First class environments and first class macros can be done. The performance constraints of the past have been removed by hardware progress and to a lesser extent by progress in compiler design.
- deleted 3y ago
  [deleted]
- kazinator 3y ago
  > Macros achieve a limited version of a fexpr with much more complicated semantics Neither of these ideas rings true. While there are things that fexprs can do that macros cannot, the reverse is also true. Macros have the opportunity to work with the source code before it is executed. Macros can issue diagnostics. Also, macros have access to the compilation environment, which can be an entirely separate machine from the run-time environment. Macros can bring in materials from auxiliary files, which would not be found on the target. A system build on fexprs rather than macros should incorporate the concept of static checking functions. For each fexpr, two additional functions should be required: - a function which knows how to statically code-walk the form which is the target of the fexpr. Perhaps of this form: (lambda (form recurse-fn) ;; function identifies, inside form, all expressions that are ;; forms and calls (recurse-fn subform) for each such subform ) - a function which knows how to check the form for errors, so the programmer can be informed about misuses of the fexpr before it is actually called on the target system. Unhygienic macros do not have complicated semantics. However, it should be acknowledged that hygiene is more naturally achieved in fexprs without effort: the fexpr's own local variables and functions are very clearly separated from evaluations of argument material, that being done by an explicit eval call, using the correct environment passed into the fexpr. A macro's local variables are likewise clearly separated, but those are expansion time. A macro has to inject new variables into the generated code which a fexpr doesn't have to do: a fexpr can use its own local variables to hold run-time temporaries related to the calculation, whereas a macro cannot do that: for run-time temporaries, it has to generate code, combining that with the argument material. Those temporaries are then in the same scope, and have to be gensyms.
  - JonChesterfield 3y ago
    Thank you for disagreeing. The relative complexity of macros vs fexpr is interesting and your reply uncovered unspoken assumptions in my thinking. Common lisp's read macros are indeed a different beast. Mutating the bytes pulled from file before parsing them is more powerful than operating on the lists of symbols. Macros imply a phase separation which fexpr do not. The compile time diagnostics and similar are an artifact of that phase separation, not of evaluation sequencing. If one wants to build in that phase separation, I think it follows that unhygienic macros are the simple option. Hygienic macros still carry much implementation complexity. If one wants hygienic behaviour, fexpr do that exactly as easily as lexically scoped lambdas, because they literally are lambda which pass the argument forms unevaluated. I implemented that as a bool tag on the function affecting calling convention - do you map eval across the arguments or not during the function call - and that's sufficient. If the desired behaviour is variable capture from the calling environment, it takes more effort from a fexpr than from a macro. You write (eval env 'foo) instead of ,foo or similar. I don't see a particular need for a code walker - in the simple case the argument gets passed around a little then evaluated, in the complicated case you need to walk the arguments the same way a macro would want to. The phase separation thing I had forgotten about. As in warning messages during compilation so that you have invariants at runtime. Macros fit that model better. Fexpr and compilation are an uneasy combination.

Macros achieve a limited version of a fexpr with much more complicated semantics. Hygienic macros take that a step further - you get implicit renaming of variables which is generally convenient - in exchange for even more complicated semantics. The abstraction leaks as well. `(reduce and (list #t #t #f))` is probably an error. Calling a macro and calling a function looks the same and sometimes is the same and sometimes falls over. Macros are sacrosanct to lisp. Hygienic macros are a crown jewel of scheme. They're still the wrong thing though. In the beginning there was dynamic scope, possibly by accident. There were also fexpr - pass the unevaluated data and the environment, instead of passing evaluated data, which were also dynamically scoped. Sometimes called nlambda. That was difficult to program with and very difficult to compile. Macros are easier to use than dynamically scoped fexpr. They're much easier to compile and lisp was getting a bit of a kicking for being too slow. There were some papers written, macros were the better choice, and here we are. Lexical scoping turned up a little while after macros won the fight. Lexically scoped fexpr with first class environments are the right thing. Simpler and more capable than macros. First class environments being another thing sacrificed on the alter of performance decades ago. Shutt noticed this and wrote about it at length. It's slightly subtle that a non-hygienic macro is a subset of fexpr. Force inline it at the caller, refuse to run various calls (e.g. pass it to functions), wrap the return value in an eval in the caller environment. A hygienic macro has to do reflective symbol renaming stuff which is also expressible as a fexpr by implementing the symbol renaming rules of scheme, which thus far I don't have the patience for. Qualified as minor, in that the wrong thing we have is still very useful. Disaster in the sense that we could have fexpr if history had happened in a slightly different sequence. Opportunity in that lisp can be better. First class environments and first class macros can be done. The performance constraints of the past have been removed by hardware progress and to a lesser extent by progress in compiler design.

deleted 3y ago
[deleted]
kazinator 3y ago
> Macros achieve a limited version of a fexpr with much more complicated semantics Neither of these ideas rings true. While there are things that fexprs can do that macros cannot, the reverse is also true. Macros have the opportunity to work with the source code before it is executed. Macros can issue diagnostics. Also, macros have access to the compilation environment, which can be an entirely separate machine from the run-time environment. Macros can bring in materials from auxiliary files, which would not be found on the target. A system build on fexprs rather than macros should incorporate the concept of static checking functions. For each fexpr, two additional functions should be required: - a function which knows how to statically code-walk the form which is the target of the fexpr. Perhaps of this form: (lambda (form recurse-fn) ;; function identifies, inside form, all expressions that are ;; forms and calls (recurse-fn subform) for each such subform ) - a function which knows how to check the form for errors, so the programmer can be informed about misuses of the fexpr before it is actually called on the target system. Unhygienic macros do not have complicated semantics. However, it should be acknowledged that hygiene is more naturally achieved in fexprs without effort: the fexpr's own local variables and functions are very clearly separated from evaluations of argument material, that being done by an explicit eval call, using the correct environment passed into the fexpr. A macro's local variables are likewise clearly separated, but those are expansion time. A macro has to inject new variables into the generated code which a fexpr doesn't have to do: a fexpr can use its own local variables to hold run-time temporaries related to the calculation, whereas a macro cannot do that: for run-time temporaries, it has to generate code, combining that with the argument material. Those temporaries are then in the same scope, and have to be gensyms.
- JonChesterfield 3y ago
  Thank you for disagreeing. The relative complexity of macros vs fexpr is interesting and your reply uncovered unspoken assumptions in my thinking. Common lisp's read macros are indeed a different beast. Mutating the bytes pulled from file before parsing them is more powerful than operating on the lists of symbols. Macros imply a phase separation which fexpr do not. The compile time diagnostics and similar are an artifact of that phase separation, not of evaluation sequencing. If one wants to build in that phase separation, I think it follows that unhygienic macros are the simple option. Hygienic macros still carry much implementation complexity. If one wants hygienic behaviour, fexpr do that exactly as easily as lexically scoped lambdas, because they literally are lambda which pass the argument forms unevaluated. I implemented that as a bool tag on the function affecting calling convention - do you map eval across the arguments or not during the function call - and that's sufficient. If the desired behaviour is variable capture from the calling environment, it takes more effort from a fexpr than from a macro. You write (eval env 'foo) instead of ,foo or similar. I don't see a particular need for a code walker - in the simple case the argument gets passed around a little then evaluated, in the complicated case you need to walk the arguments the same way a macro would want to. The phase separation thing I had forgotten about. As in warning messages during compilation so that you have invariants at runtime. Macros fit that model better. Fexpr and compilation are an uneasy combination.