17 ms·

Defense of Lisp macros: The automotive field as a case in point

227 points by molteanu 2y ago

deleted 2y ago
[deleted]
pfdietz 2y ago
> "I don't think this is truly realized, but all this information written down in word sheets, requirements, knowledge passed on in meetings, little tricks and short hacks solved in video calls, information passed in emails and instant messaging is information that is part of the project, that is put down in some kind of a language, a non-formal one like English or tables or boxes or scribbles or laughs and jokes. The more a project has of these, the harder it is to talk about and verify the actual state of the project as it is obviously impossible to evaluate a non-formal language like you would a formal programming language and thus it becomes that much harder to explore and play with, test and explore the project." I've thought all this sloppy side information would be a good target for the currently in vogue AI techniques.
- molteanu 2y ago
  That would only mean we'd add even more tools on top of the tools that currently sit themselves on top of C (in this particular case) to fix our 'sloppy' ways, no?
  - pfdietz 2y ago
    Yes, but the hope would be a more general tool rather than a raft of specialized tools.
FrustratedMonky 2y ago
So how would we go about switching the entire software industry to use LISP more? I've been struggling with this idea for awhile. It seems that the best languages don't get adopted. The only consistent explanation I've seen that it is about 'easy'. The other languages have tools to make them easy, easy IDE's, the languages 'solve' one 'thing' and using them for that 'one thing' is easier to than building your own in LISP. You can do anything with LISP, sure, but there is a learning curve, lot of 'ways of thinking' to adopt the brain to in order to solve problems. Personally, I do wish we could somehow re-vamp the CS Education system to focus on LISP and other ML languages, and train more for the thinking process, not just how to connect up some Java Libraries.
- adamddev1 2y ago
  There was a great episode of Type Theory for All with Conal Elliot as a guest where he decried MITs decision to switch to teaching Python instead of Scheme for introductory programming. He makes a very good case for how CS education should be aiming to pursue the greatest truths and not just equipping people with whatever employers want.
  - anta40 2y ago
    A podcast about type theory and other programming language design topics? Oh well auto subscribe :)
- dartos 2y ago
  Imo the issue is undergrad cs programs have an identity crisis. Are they the entry into the world of computer science. Should they teach set theory, computer organization, language theory, etc. OR are they trying to prepare the new wave of workers. Should they teach protocols like http, industry tools like Java and python, and good test practices? A well rounded engineer should have a grasp on it all, but with only 4 years, what will attract more funding and more students?
  - VyseofArcadia 2y ago
    I've been advocating for years that CS departments need to bifurcate. Actual CS classes, the theory-heavy kind, need to be reabsorbed into math departments. People who want to study computer science academically would be better served by getting a math degree. The "how to write software" courses need be absorbed into engineering departments, and maybe the extra discipline gains from actual engineers teaching these courses can start turning software engineers from computer programmers with an inflated job title into actual engineers. Students can then make a choice. Do I get a degree in computer science and be better prepared for academia, essentially being a mathematician who specializes in computation? Or do I get a degree in computer engineering and be better prepared to write reliable software in industry? Of course this distinction exists today, and some universities do offer separate CS and CE degrees, but in practice it seems more often to be smashed together into a catch-all CS degree that may or may not be more theory or practice focused depending on the program.
roenxi 2y ago
In my experience (Clojure) macro-heavy libraries tend to be powerful but brittle. A lisp programmer can do things in a library that non-lisp programmers can't realistically attempt (Typed Clojure, for example). But the trade offs are very real though hard to put a finger on. There are several Clojure libraries that are cool and completely reliant on macros to be implemented. Eventually, good Clojure programmers seem to stop using them because the trade-offs are worse error messages or uncomfortable edge-case behaviours. The base language is just so good that it doesn't really matter, but I've had several experiences where the theoretically-great library had to be abandoned because of how it interacts with debugging tools and techniques. It isn't the macros fault, they just hint that a programmer is about to do something that, when it fails, fails in a way that is designed in to the system and can't be easily worked around. Macros are basically for creating new syntax constructs - great but when the syntax has bugs, the programmer has a problem. And the community tooling probably won't understand it.
- PaulHoule 2y ago
  For a few years I've tried some crazy experiments towards metaprogramming in Java https://github.com/paulhoule/ferocity https://github.com/paulhoule/ferocity my plan there was to write ferocity0 in Java that is able to stub out the standard library and then write ferocity1 in ferocity0 (and maybe a ferocity2 in ferocity1) so that I don't have to write repetitive code to stub out all the operators, the eight primitive data types, etc. I worked out a way to write out Java code in a form that looks like S-expressions https://github.com/paulhoule/ferocity/blob/main/ferocity0/src/test/java/com/ontology2/ferocity/TestFierceStrings.java https://github.com/paulhoule/ferocity/blob/main/ferocity0/sr... which bulks up the code even more than ordinary Java so to make up for it I want to use "macros" aggressively which might get the code size reasonable but I'm sure people would struggle to understand it. I switched to other projects but yesterday I was working on some Java that had a lot of boilerplate and was thinking it ought to be possible to do something with compile time annotations along the lines of @CompileTime static void generate(LiveClass that) { that.addMethod("methodName",... arguments ...,... body...) } and even write a function like that which might see a class annotation @Boilerplate("argument") and add final static PARAM_ARGUMENT = "${argument}" or something like that.
VyseofArcadia 2y ago
I was a bit baffled the first time I was introduced to Simulink Coder. I get the value proposition: simulate your model so that you have confidence it's doing the right thing, and then essentially just run that model as code instead of implementing it yourself and possibly introducing mistakes. What I didn't understand, as a software guy, not an engineering guy, was why on earth you'd want a graphical programming language to do that. Surely it would be easier to just write your model in a traditional text-based language. Hell, the same company even makes their own language (MATLAB) that would be perfect for the task. I did a little digging, and it turns out I had it backwards. It's not that Simulink invented a new graphical programming language to do this stuff. Control systems engineers have been studying and documenting systems using Simulink-style block diagrams since at least the 1960s. Simulink just took something that used to be a paper and chalkboard modeling tool and made it executable on a computer.
- jocaal 2y ago
  The people using matlab are typically experts in things other than programming. Matlab is for people designing antennas or control systems, where the math is more important than the code.
  - VyseofArcadia 2y ago
    Most definitely, but my point was to my programmer-brain, nearly any text-based modeling language would be better than some sort of awkward graphical diagram. And since the same company makes MATLAB, why not? It turns out non-programmers actually like graphical tools. Go figure.
    - molteanu 2y ago
      I've linked this thread from HN in the article about visual programming, if you find it useful or insightful, https://news.ycombinator.com/item?id=14484244 https://news.ycombinator.com/item?id=14484244
    - jocaal 2y ago
      > It turns out non-programmers actually like graphical tools. Go figure. I'd rather say that visual tools are better abstractions than what text based tools can provide for those specific scenarios. Some problems are easier solved with diagrams.
BaculumMeumEst 2y ago
Macros are one of the widely touted superpowers of lisp. And yet in Clojure, which is the most widely used dialect in modern use, it is idiomatic to avoid macros as much as possible. None of the claimed advantages really hold up under scrutiny.
- rscho 2y ago
  > idiomatic to avoid macros as much as possible. Same in all lisps. The advantages of having macros vary on a case-by-case basis, but one thing is quite sure: the number of programmers collaborating on a project is inversely proportional to how much you should use macros. For a solo dev, especially when doing science-y stuff, they're invaluable.
  - slaymaker1907 2y ago
    That's only true of macros created for the purpose of reducing boilerplate code. However, the best macros actually reduce errors and improve code quality by enforcing invariants more than can be done without macros.
- bjoli 2y ago
  Many people use macros as a poor man's inliner, to generate code in a way that adds no benefits over a good compiler. You rarely actually need syntactic abstractions, but when you do you need macros.
- wtetzner 2y ago
  > it is idiomatic to avoid macros as much as possible. Of course, don't use a macro if you don't need one. But when you do hit a case where you need one, it's better than using/writing external tooling to solve the problem. > None of the claimed advantages really hold up under scrutiny. This doesn't follow.
- mst 2y ago
  Macros are an escape hatch. "Don't use a macro unless you need to" is good advice, but when you -do- need to, having macros around is lovely. Yes, you can achieve the same thing by typing out what would've been the expansion yourself, but in cases where macros are a good idea the expansion would've been sufficiently prone to mistakes when tweaking it that maintaining the macro is an improvement.
codr7 2y ago
Macros are more abstract, one meta level up, hence errors are more difficult to relate to code and reason about. They are also more powerful, so errors can have dramatic consequences. Any kind of code generation setup will have the same characteristics. Sloppy macros clashing with local names can lead to pretty long debug sessions, but it's the first thing you learn to avoid. And you're basically inventing your own ad hoc programming language to some extent, the effort spent on error handling tend to not live up to those requirements. All of that being said, I wouldn't trade them for anything, and I haven't seen any convincing alternatives to Lisp for the full experience.
- kazinator 2y ago
  > one meta level up Which level is that? Is cond higher than if? Which one is the macro, again?
  - codr7 2y ago
    When you write macros, you're working one meta level up, since you're writing code that generates code (that generates code etc, but one level higher than usual). From a user perspective it's different. Macros are more limited; not first class and can't be passed around and called like functions.
  - shadowgovt 2y ago
    If we're talking LISP, I believe "if" is a form, not a macro (because it breaks evaluation rules by only evaluating one branch or the other at runtime depending on the evaluation of the conditional).
    - kazinator 2y ago
      In fact, MacCarthy invented a multi-branch conditional construct which in M-expressions looked like [test1 -> value1; test2 -> value2; T -> default] The S-expression form was cond: (cond (test1 value1) (test2 value2) (T default)) The if macro came later, defined in terms of cond. Macros are no operationally distinguishable from special operators. You know that a form is a macro because there is a binding for the symbol as a macro, and you can ask for the expansion. Because macros expand to code that may contain special forms that control evaluation, macros thereby exhibit control over evaluation.
codr7 2y ago
On that subject, I once made a serious effort at explaining why Lisp macros are so useful: https://github.com/codr7/whirlisp https://github.com/codr7/whirlisp
pfdietz 2y ago
Without Lisp-like macros, you need preprocessors or code generators. Something like Yacc/Bison is easily done with suitable macros. In Common Lisp, macros also enable a kind of Aspect Oriented Programming. That's because one can dynamically modify macro expansion using the macroexpand hook. It's a way to modify a code base without changing the source files, which has all sorts of uses. http://clhs.lisp.se/Body/v_mexp_h.htm#STmacroexpand-hookST http://clhs.lisp.se/Body/v_mexp_h.htm#STmacroexpand-hookST One can implement things natively in Common Lisp, like code coverage tools, that require separate tooling in other languages.
- kazinator 2y ago
  > one can dynamically modify macro expansion using the macroexpand hook That's an incredibly bad idea, and it's not dynamic except in purely interpreted Lisps that expand the same piece of code each time it is executed. The hooks is absolutely global; whatever you put in there must not break anything. From the spec itself: "For this reason, it is frequently best to confine its uses to debugging situations."
  - pfdietz 2y ago
    One would bind the macroexpand hook variable around the call to COMPILE or COMPILE-FILE for that code. The binding goes away after that. It wouldn't be global at all.
    - froh 2y ago
      yah dynamic binding id another thing I dearly miss. thread local is like global in that it doesn't support for partial stacks of coroutines or generators. I'd love to have some dynamic _call stack based_ context lookup in languages like Python or c++.
      - gpderetta 2y ago
        With RAII, it is not terribly hard to build dynamic scoping on top of thread_local variables in C++. template<class T> struct scoped_replace { scoped_replace(T& ref, T new_val) : ref(ref), old_val(ref) { ref = std::move(new_val); } ~scoped_replace() { ref = std::move(old_val); } T& ref; T old_val; }; thread_local int val = 0; int foo() { return val; } int bar() { scoped_replace _(val, 42); return foo(); } int main() { foo(); // returns 0 bar(); // returns 42; foo(); // still returns 0 }
cryptonector 2y ago
> Replacing Lisp's beautiful parentheses with dozens of special tools and languages, none powerful enough to conquer the whole software landscape, leads to fragmentation and extra effort from everyone, vendors and developers alike. The automotive field is a case in point. This completely ignores Haskell.
- tsimionescu 2y ago
  Haskell is a niche language, not widely used in any industry, and it has nothing like the power and tooling of Lisp. What would be its place in this discussion?
- bunderbunder 2y ago
  Last I checked, the closest Haskell gets to first-class metaprogramming is compiler extensions.
qohen 2y ago
BTW, for anyone interested in learning more about Lisp macros, Paul Graham's book about advanced Lisp programming, On Lisp, covers the topic pretty extensively and it's freely downloadable from his website: Book description: https://paulgraham.com/onlisp.html https://paulgraham.com/onlisp.html Download page: https://paulgraham.com/onlisptext.html https://paulgraham.com/onlisptext.html
- ocodo 2y ago
  The Giga Monkeys Book, Practical Common Lisp is also excellent: https://gigamonkeys.com/book/ https://gigamonkeys.com/book/
- lkuty 2y ago
  And then LoL: https://letoverlambda.com/ https://letoverlambda.com/ I started to read the book years ago but it was too wild for me. I should give it another try. One of the so many things I have to read.
djaouen 2y ago
I am not reading 10,000+ words on why macros are good. I know macros are good because smart people tell me they are. There is also plentiful evidence of macros being useful to programmers. Who are you trying to convince here?
- molteanu 2y ago
  I am writing for myself. I've also worked in the automotive industry as a consultant for a few years and I've seen the effects of these extra tools I'm mentioning in the article on me and my colleagues and the industry as a whole. I've gathered all the observations in one place. It has been a nice experience for me writing it, it has cleared up some ideas, brought to life new ones, as it always happens when you think you see something very clearly only to change your opinion and see new angles once you put it in words or once you try to articulate it. I am glad if it's useful to others as what others write and wrote over the years, including about Lisp, was useful to me. It's just free food, if you will. They've charged nothing for it, even if they worked for years on producing it, I'm grateful for that and now I'm returning the favor. If it's good food or not, I can't say. If you don't like it, leave it to others, no harm in that.
jancsika 2y ago
> A confirmation of Greenspun's Tenth Rule, if you will. I've always found this rule suspect. If it were true you'd have had lispers loudly reimplementing well-known, slowly-and-buggy mid-size C programs in common lisp, at roughly the rateand loudness the Rust people reimplement old C programs in Rust. Did that actually happen? If not, this has to be the greatest expression of envy in all of programming language lore.
- molteanu 2y ago
  No, it just means you can't avoid the reality and the complexity of the problems you'll be dealing with and you won't be able to tackle them unless you bring in more tools or programming languages, be them visual or special languages invented just for the purpose, or some clever way to use excel sheets and generate code from that, for example. Or it means that, yes, you can choose the easier language, the more intuitive one instead of the abstract beasts and it all goes well for a while. But eventually you'll have to go past the "they did this and then they did that", past the simple to understand if's and else's of everyday life. Then you'll realize that yes, we need this special extra feature here, this clever way of doing things there and before you know it, you've developed dozens of tools, standards and languages just to make up for the lack of power of your initial language. Or you could have chosen the harder language, harder to learn and wield, that is, and it would have offered you better abstractions, it would have offered you better means to develop your ideas and put them into practice without you needing to invent your own tooling. That's how I've seen it happen in practice. I've expanded it in this article, but it came the other way round. I had a feeling that all this tooling is actually unnecessary or just the result of having bad tooling to start with. Once I've developed these ideas I've remembered about Greenspun's Tenth Rule that I've read and heard about all these years without quite understanding it 100%. Now I think I do. You have to see it with your own eyes, though. If you read it once and it doesn't make sense, re-reading it probably wouldn't. I hope this helps.
  - brabel 2y ago
    I think the article was pretty convincing. Even painfully so as it went through the atrocities people are doing in this particular industry to achieve things that, you're completely right, would be trivial in Lisp. Unfortunately, most critique you're getting here is coming from people who didn't read the article, as you can see by the lack of comments addressing any specific part of the article.
RealityVoid 2y ago
IMO, someone should kill XCP and A2L. They are in effect, a semihosted debugger. I would rather have semihosted gdb and use .elf. The tooling around A2L is horrendous.
- molteanu 2y ago
  Well, yes! I'm trying my best to make your wish come true by spreading the word. Good to know I'm not the only one finding it a bit too complicated and sad. Do share your experience with it, if you will.
continuational 2y ago
Macros allow you to bring highly tailored DSLs into your code base. You can practice language oriented programming. It's seductive, but there's a steep cost. You bring in a bunch of new notation, which you have to learn and teach and remember. You have to consider all the interactions that happen when you mix notations. You have to implement all the tooling. Usually what happens is you don't quite remember what things mean after a while, and you have to study it again. You didn't consider all the interactions, so it isn't that well behaved. And you never got around to implement quality tooling - and there's no obvious place to plug that tooling in, either. And then it turns out that the notation wasn't quite right for implementing the changing requirements of the business.
agumonkey 2y ago
Wow this starts strong: > Replacing Lisp's beautiful parentheses with dozens of special tools and languages, none powerful enough to conquer the whole software landscape, leads to fragmentation and extra effort from everyone, vendors and developers alike. The automotive field is a case in point. This has been a struggle of mine since day one. Coming from the java world with more formats, tools, IDEs everytime people create a problem was absurd. Lisp was all absorbing.. much reuse on all metalevels.. yet sometimes I believe a bit of "organ"ism is useful.
bunderbunder 2y ago
The book Beautiful Racket is the best defense of macros out there, on "show, don't tell" grounds. Also, it's just so well-written and describes such an interesting set of tools that I'd argue it's worth a skim even if you don't intend to learn the language. https://beautifulracket.com/ https://beautifulracket.com/
whalesalad 2y ago
For such a long and detailed post, not a single lisp expression or macro is visible in this post.
deterministic 2y ago
I am not a fan of Lisp. However I sure wish all command line tools would use the same Lisp dialect for scripting. Instead of each Linux tool inventing their own awful, badly documented, makes-you-want-to-vomit bad joke of a scripting language.

227 points by molteanu 2y ago

deleted 2y ago

[deleted]

> "I don't think this is truly realized, but all this information written down in word sheets, requirements, knowledge passed on in meetings, little tricks and short hacks solved in video calls, information passed in emails and instant messaging is information that is part of the project, that is put down in some kind of a language, a non-formal one like English or tables or boxes or scribbles or laughs and jokes. The more a project has of these, the harder it is to talk about and verify the actual state of the project as it is obviously impossible to evaluate a non-formal language like you would a formal programming language and thus it becomes that much harder to explore and play with, test and explore the project." I've thought all this sloppy side information would be a good target for the currently in vogue AI techniques.

molteanu 2y ago
That would only mean we'd add even more tools on top of the tools that currently sit themselves on top of C (in this particular case) to fix our 'sloppy' ways, no?
- pfdietz 2y ago
  Yes, but the hope would be a more general tool rather than a raft of specialized tools.

FrustratedMonky 2y ago

So how would we go about switching the entire software industry to use LISP more? I've been struggling with this idea for awhile. It seems that the best languages don't get adopted. The only consistent explanation I've seen that it is about 'easy'. The other languages have tools to make them easy, easy IDE's, the languages 'solve' one 'thing' and using them for that 'one thing' is easier to than building your own in LISP. You can do anything with LISP, sure, but there is a learning curve, lot of 'ways of thinking' to adopt the brain to in order to solve problems. Personally, I do wish we could somehow re-vamp the CS Education system to focus on LISP and other ML languages, and train more for the thinking process, not just how to connect up some Java Libraries.

adamddev1 2y ago
There was a great episode of Type Theory for All with Conal Elliot as a guest where he decried MITs decision to switch to teaching Python instead of Scheme for introductory programming. He makes a very good case for how CS education should be aiming to pursue the greatest truths and not just equipping people with whatever employers want.
- anta40 2y ago
  A podcast about type theory and other programming language design topics? Oh well auto subscribe :)
dartos 2y ago
Imo the issue is undergrad cs programs have an identity crisis. Are they the entry into the world of computer science. Should they teach set theory, computer organization, language theory, etc. OR are they trying to prepare the new wave of workers. Should they teach protocols like http, industry tools like Java and python, and good test practices? A well rounded engineer should have a grasp on it all, but with only 4 years, what will attract more funding and more students?
- VyseofArcadia 2y ago
  I've been advocating for years that CS departments need to bifurcate. Actual CS classes, the theory-heavy kind, need to be reabsorbed into math departments. People who want to study computer science academically would be better served by getting a math degree. The "how to write software" courses need be absorbed into engineering departments, and maybe the extra discipline gains from actual engineers teaching these courses can start turning software engineers from computer programmers with an inflated job title into actual engineers. Students can then make a choice. Do I get a degree in computer science and be better prepared for academia, essentially being a mathematician who specializes in computation? Or do I get a degree in computer engineering and be better prepared to write reliable software in industry? Of course this distinction exists today, and some universities do offer separate CS and CE degrees, but in practice it seems more often to be smashed together into a catch-all CS degree that may or may not be more theory or practice focused depending on the program.

roenxi 2y ago

In my experience (Clojure) macro-heavy libraries tend to be powerful but brittle. A lisp programmer can do things in a library that non-lisp programmers can't realistically attempt (Typed Clojure, for example). But the trade offs are very real though hard to put a finger on. There are several Clojure libraries that are cool and completely reliant on macros to be implemented. Eventually, good Clojure programmers seem to stop using them because the trade-offs are worse error messages or uncomfortable edge-case behaviours. The base language is just so good that it doesn't really matter, but I've had several experiences where the theoretically-great library had to be abandoned because of how it interacts with debugging tools and techniques. It isn't the macros fault, they just hint that a programmer is about to do something that, when it fails, fails in a way that is designed in to the system and can't be easily worked around. Macros are basically for creating new syntax constructs - great but when the syntax has bugs, the programmer has a problem. And the community tooling probably won't understand it.

PaulHoule 2y ago
For a few years I've tried some crazy experiments towards metaprogramming in Java https://github.com/paulhoule/ferocity https://github.com/paulhoule/ferocity my plan there was to write ferocity0 in Java that is able to stub out the standard library and then write ferocity1 in ferocity0 (and maybe a ferocity2 in ferocity1) so that I don't have to write repetitive code to stub out all the operators, the eight primitive data types, etc. I worked out a way to write out Java code in a form that looks like S-expressions https://github.com/paulhoule/ferocity/blob/main/ferocity0/src/test/java/com/ontology2/ferocity/TestFierceStrings.java https://github.com/paulhoule/ferocity/blob/main/ferocity0/sr... which bulks up the code even more than ordinary Java so to make up for it I want to use "macros" aggressively which might get the code size reasonable but I'm sure people would struggle to understand it. I switched to other projects but yesterday I was working on some Java that had a lot of boilerplate and was thinking it ought to be possible to do something with compile time annotations along the lines of @CompileTime static void generate(LiveClass that) { that.addMethod("methodName",... arguments ...,... body...) } and even write a function like that which might see a class annotation @Boilerplate("argument") and add final static PARAM_ARGUMENT = "${argument}" or something like that.

VyseofArcadia 2y ago

I was a bit baffled the first time I was introduced to Simulink Coder. I get the value proposition: simulate your model so that you have confidence it's doing the right thing, and then essentially just run that model as code instead of implementing it yourself and possibly introducing mistakes. What I didn't understand, as a software guy, not an engineering guy, was why on earth you'd want a graphical programming language to do that. Surely it would be easier to just write your model in a traditional text-based language. Hell, the same company even makes their own language (MATLAB) that would be perfect for the task. I did a little digging, and it turns out I had it backwards. It's not that Simulink invented a new graphical programming language to do this stuff. Control systems engineers have been studying and documenting systems using Simulink-style block diagrams since at least the 1960s. Simulink just took something that used to be a paper and chalkboard modeling tool and made it executable on a computer.

jocaal 2y ago
The people using matlab are typically experts in things other than programming. Matlab is for people designing antennas or control systems, where the math is more important than the code.
- VyseofArcadia 2y ago
  Most definitely, but my point was to my programmer-brain, nearly any text-based modeling language would be better than some sort of awkward graphical diagram. And since the same company makes MATLAB, why not? It turns out non-programmers actually like graphical tools. Go figure.
  - molteanu 2y ago
    I've linked this thread from HN in the article about visual programming, if you find it useful or insightful, https://news.ycombinator.com/item?id=14484244 https://news.ycombinator.com/item?id=14484244
  - jocaal 2y ago
    > It turns out non-programmers actually like graphical tools. Go figure. I'd rather say that visual tools are better abstractions than what text based tools can provide for those specific scenarios. Some problems are easier solved with diagrams.

BaculumMeumEst 2y ago

Macros are one of the widely touted superpowers of lisp. And yet in Clojure, which is the most widely used dialect in modern use, it is idiomatic to avoid macros as much as possible. None of the claimed advantages really hold up under scrutiny.

rscho 2y ago
> idiomatic to avoid macros as much as possible. Same in all lisps. The advantages of having macros vary on a case-by-case basis, but one thing is quite sure: the number of programmers collaborating on a project is inversely proportional to how much you should use macros. For a solo dev, especially when doing science-y stuff, they're invaluable.
- slaymaker1907 2y ago
  That's only true of macros created for the purpose of reducing boilerplate code. However, the best macros actually reduce errors and improve code quality by enforcing invariants more than can be done without macros.
bjoli 2y ago
Many people use macros as a poor man's inliner, to generate code in a way that adds no benefits over a good compiler. You rarely actually need syntactic abstractions, but when you do you need macros.
wtetzner 2y ago
> it is idiomatic to avoid macros as much as possible. Of course, don't use a macro if you don't need one. But when you do hit a case where you need one, it's better than using/writing external tooling to solve the problem. > None of the claimed advantages really hold up under scrutiny. This doesn't follow.
mst 2y ago
Macros are an escape hatch. "Don't use a macro unless you need to" is good advice, but when you -do- need to, having macros around is lovely. Yes, you can achieve the same thing by typing out what would've been the expansion yourself, but in cases where macros are a good idea the expansion would've been sufficiently prone to mistakes when tweaking it that maintaining the macro is an improvement.

codr7 2y ago

Macros are more abstract, one meta level up, hence errors are more difficult to relate to code and reason about. They are also more powerful, so errors can have dramatic consequences. Any kind of code generation setup will have the same characteristics. Sloppy macros clashing with local names can lead to pretty long debug sessions, but it's the first thing you learn to avoid. And you're basically inventing your own ad hoc programming language to some extent, the effort spent on error handling tend to not live up to those requirements. All of that being said, I wouldn't trade them for anything, and I haven't seen any convincing alternatives to Lisp for the full experience.

kazinator 2y ago
> one meta level up Which level is that? Is cond higher than if? Which one is the macro, again?
- codr7 2y ago
  When you write macros, you're working one meta level up, since you're writing code that generates code (that generates code etc, but one level higher than usual). From a user perspective it's different. Macros are more limited; not first class and can't be passed around and called like functions.
- shadowgovt 2y ago
  If we're talking LISP, I believe "if" is a form, not a macro (because it breaks evaluation rules by only evaluating one branch or the other at runtime depending on the evaluation of the conditional).
  - kazinator 2y ago
    In fact, MacCarthy invented a multi-branch conditional construct which in M-expressions looked like [test1 -> value1; test2 -> value2; T -> default] The S-expression form was cond: (cond (test1 value1) (test2 value2) (T default)) The if macro came later, defined in terms of cond. Macros are no operationally distinguishable from special operators. You know that a form is a macro because there is a binding for the symbol as a macro, and you can ask for the expansion. Because macros expand to code that may contain special forms that control evaluation, macros thereby exhibit control over evaluation.

codr7 2y ago

On that subject, I once made a serious effort at explaining why Lisp macros are so useful: https://github.com/codr7/whirlisp https://github.com/codr7/whirlisp

pfdietz 2y ago

Without Lisp-like macros, you need preprocessors or code generators. Something like Yacc/Bison is easily done with suitable macros. In Common Lisp, macros also enable a kind of Aspect Oriented Programming. That's because one can dynamically modify macro expansion using the macroexpand hook. It's a way to modify a code base without changing the source files, which has all sorts of uses. http://clhs.lisp.se/Body/v_mexp_h.htm#STmacroexpand-hookST http://clhs.lisp.se/Body/v_mexp_h.htm#STmacroexpand-hookST One can implement things natively in Common Lisp, like code coverage tools, that require separate tooling in other languages.

kazinator 2y ago
> one can dynamically modify macro expansion using the macroexpand hook That's an incredibly bad idea, and it's not dynamic except in purely interpreted Lisps that expand the same piece of code each time it is executed. The hooks is absolutely global; whatever you put in there must not break anything. From the spec itself: "For this reason, it is frequently best to confine its uses to debugging situations."
- pfdietz 2y ago
  One would bind the macroexpand hook variable around the call to COMPILE or COMPILE-FILE for that code. The binding goes away after that. It wouldn't be global at all.
  - froh 2y ago
    yah dynamic binding id another thing I dearly miss. thread local is like global in that it doesn't support for partial stacks of coroutines or generators. I'd love to have some dynamic _call stack based_ context lookup in languages like Python or c++.
    - gpderetta 2y ago
      With RAII, it is not terribly hard to build dynamic scoping on top of thread_local variables in C++. template<class T> struct scoped_replace { scoped_replace(T& ref, T new_val) : ref(ref), old_val(ref) { ref = std::move(new_val); } ~scoped_replace() { ref = std::move(old_val); } T& ref; T old_val; }; thread_local int val = 0; int foo() { return val; } int bar() { scoped_replace _(val, 42); return foo(); } int main() { foo(); // returns 0 bar(); // returns 42; foo(); // still returns 0 }

cryptonector 2y ago

> Replacing Lisp's beautiful parentheses with dozens of special tools and languages, none powerful enough to conquer the whole software landscape, leads to fragmentation and extra effort from everyone, vendors and developers alike. The automotive field is a case in point. This completely ignores Haskell.

tsimionescu 2y ago
Haskell is a niche language, not widely used in any industry, and it has nothing like the power and tooling of Lisp. What would be its place in this discussion?
bunderbunder 2y ago
Last I checked, the closest Haskell gets to first-class metaprogramming is compiler extensions.

qohen 2y ago

BTW, for anyone interested in learning more about Lisp macros, Paul Graham's book about advanced Lisp programming, On Lisp, covers the topic pretty extensively and it's freely downloadable from his website: Book description: https://paulgraham.com/onlisp.html https://paulgraham.com/onlisp.html Download page: https://paulgraham.com/onlisptext.html https://paulgraham.com/onlisptext.html

ocodo 2y ago
The Giga Monkeys Book, Practical Common Lisp is also excellent: https://gigamonkeys.com/book/ https://gigamonkeys.com/book/
lkuty 2y ago
And then LoL: https://letoverlambda.com/ https://letoverlambda.com/ I started to read the book years ago but it was too wild for me. I should give it another try. One of the so many things I have to read.

djaouen 2y ago

I am not reading 10,000+ words on why macros are good. I know macros are good because smart people tell me they are. There is also plentiful evidence of macros being useful to programmers. Who are you trying to convince here?

molteanu 2y ago
I am writing for myself. I've also worked in the automotive industry as a consultant for a few years and I've seen the effects of these extra tools I'm mentioning in the article on me and my colleagues and the industry as a whole. I've gathered all the observations in one place. It has been a nice experience for me writing it, it has cleared up some ideas, brought to life new ones, as it always happens when you think you see something very clearly only to change your opinion and see new angles once you put it in words or once you try to articulate it. I am glad if it's useful to others as what others write and wrote over the years, including about Lisp, was useful to me. It's just free food, if you will. They've charged nothing for it, even if they worked for years on producing it, I'm grateful for that and now I'm returning the favor. If it's good food or not, I can't say. If you don't like it, leave it to others, no harm in that.

jancsika 2y ago

> A confirmation of Greenspun's Tenth Rule, if you will. I've always found this rule suspect. If it were true you'd have had lispers loudly reimplementing well-known, slowly-and-buggy mid-size C programs in common lisp, at roughly the rateand loudness the Rust people reimplement old C programs in Rust. Did that actually happen? If not, this has to be the greatest expression of envy in all of programming language lore.

molteanu 2y ago
No, it just means you can't avoid the reality and the complexity of the problems you'll be dealing with and you won't be able to tackle them unless you bring in more tools or programming languages, be them visual or special languages invented just for the purpose, or some clever way to use excel sheets and generate code from that, for example. Or it means that, yes, you can choose the easier language, the more intuitive one instead of the abstract beasts and it all goes well for a while. But eventually you'll have to go past the "they did this and then they did that", past the simple to understand if's and else's of everyday life. Then you'll realize that yes, we need this special extra feature here, this clever way of doing things there and before you know it, you've developed dozens of tools, standards and languages just to make up for the lack of power of your initial language. Or you could have chosen the harder language, harder to learn and wield, that is, and it would have offered you better abstractions, it would have offered you better means to develop your ideas and put them into practice without you needing to invent your own tooling. That's how I've seen it happen in practice. I've expanded it in this article, but it came the other way round. I had a feeling that all this tooling is actually unnecessary or just the result of having bad tooling to start with. Once I've developed these ideas I've remembered about Greenspun's Tenth Rule that I've read and heard about all these years without quite understanding it 100%. Now I think I do. You have to see it with your own eyes, though. If you read it once and it doesn't make sense, re-reading it probably wouldn't. I hope this helps.
- brabel 2y ago
  I think the article was pretty convincing. Even painfully so as it went through the atrocities people are doing in this particular industry to achieve things that, you're completely right, would be trivial in Lisp. Unfortunately, most critique you're getting here is coming from people who didn't read the article, as you can see by the lack of comments addressing any specific part of the article.

RealityVoid 2y ago

IMO, someone should kill XCP and A2L. They are in effect, a semihosted debugger. I would rather have semihosted gdb and use .elf. The tooling around A2L is horrendous.

molteanu 2y ago
Well, yes! I'm trying my best to make your wish come true by spreading the word. Good to know I'm not the only one finding it a bit too complicated and sad. Do share your experience with it, if you will.

continuational 2y ago

Macros allow you to bring highly tailored DSLs into your code base. You can practice language oriented programming. It's seductive, but there's a steep cost. You bring in a bunch of new notation, which you have to learn and teach and remember. You have to consider all the interactions that happen when you mix notations. You have to implement all the tooling. Usually what happens is you don't quite remember what things mean after a while, and you have to study it again. You didn't consider all the interactions, so it isn't that well behaved. And you never got around to implement quality tooling - and there's no obvious place to plug that tooling in, either. And then it turns out that the notation wasn't quite right for implementing the changing requirements of the business.

agumonkey 2y ago

Wow this starts strong: > Replacing Lisp's beautiful parentheses with dozens of special tools and languages, none powerful enough to conquer the whole software landscape, leads to fragmentation and extra effort from everyone, vendors and developers alike. The automotive field is a case in point. This has been a struggle of mine since day one. Coming from the java world with more formats, tools, IDEs everytime people create a problem was absurd. Lisp was all absorbing.. much reuse on all metalevels.. yet sometimes I believe a bit of "organ"ism is useful.

bunderbunder 2y ago

The book Beautiful Racket is the best defense of macros out there, on "show, don't tell" grounds. Also, it's just so well-written and describes such an interesting set of tools that I'd argue it's worth a skim even if you don't intend to learn the language. https://beautifulracket.com/ https://beautifulracket.com/

whalesalad 2y ago

For such a long and detailed post, not a single lisp expression or macro is visible in this post.

deterministic 2y ago

I am not a fan of Lisp. However I sure wish all command line tools would use the same Lisp dialect for scripting. Instead of each Linux tool inventing their own awful, badly documented, makes-you-want-to-vomit bad joke of a scripting language.