6 ms·

Of Lisp Macros and Washing Machines

99 points by MrJagil 15y ago

agentultra 15y ago
I told a friend how I was going to implement my next project in Lisp. He immediately thought I was nuts to do so -- that it would never get acquired or it would be too hard to hire people to work on when it grows. All of the tried-and-true objections to using Lisp for a "real" project. Then I read an article like this and realize I've been explaining it all wrong. This is a great read and a real eye-opener.
- sedachv 15y ago
  Give your programmers a copy of _Practical Common Lisp_, it's not difficult to learn.
kenjackson 15y ago
What do CRUD apps have to do with Lisp macros? The points are disconnected and points are introduced with no evidence or logic to back them. The only thing I can gather is this guy is about to become a billionaire, because he has found a way to rid the world of spreadsheets and CRUD apps... apparently with steel mills.
- troutwine 15y ago
  I try to think the best of people. With no code to Loper OS that I can find and a writing style that perpetuates lisp elitism using peculiar analogies to the Industrial Revolution without any evidence that this elitism is warrented--I use lisp, I like lisp but it's one tool in a bag--I find it hard to believe that Mr. Datskovskiy is serious. Surely this is brilliant satire?
  - asciilifeform 15y ago
    > without any evidence that this elitism is warranted What kind of evidence would you accept? And if you won't accept it from R. Gabriel, P. Graham, P. Greenspun, and other "household" names, why should the rest of us bother sweating to assemble a watertight case for your wastebasket's eyes?
    - troutwine 15y ago
      > What kind of evidence would you accept? Code. Every author takes on himself the onerous to justify his claims. I am familiar with the works of the authors you mention; they present a good argument that lisp is a fine language for fine works. It is an entirely different argument to assert that, merely through the use of lisp, you will revolutionize the world. That, as I read it, is the claim made by the original author. It is that claim which has gone unsubstantiated, for years. The rest of you--incidentally, no need to use exclusionary language: we are all friend here--need do nothing. Mr. Datskovskiy has some code to deliver. A bootloader in two years is no accomplishment, though Mr. Datskovskiy has excelled in punditry in this period.
      - asciilifeform 15y ago
        Original author speaking. If you actually read the blog (unbearable ordeal, I understand) you would know that Lisp (in available incarnations on available hardware) is not enough: http://www.loper-os.org/?p=284 http://www.loper-os.org/?p=284 Let's see how much of an OS and hardware architecture you can design in several years - without using a single line of code written by another person. Starting with the NAND gate - because that is where you will have to start if you truly intend to avoid recapitulating the mistakes of the past three decades. There was no sequel to the bootloader - I have been occupied with reverse-engineering the bitstream format of a major brand of FPGA (no naming names) in order to dispense with the proprietary x86 toolchain (and eventually with the abomination called Verilog.) Publishing incremental progress in bite-sized chunks is vastly over-rated. It wastes everyone's time and contributes to the proliferation of hideously ragtag systems reminiscent of "Junkyard Wars" (aka much of the software industry.) Punditry, on the other hand, is a relaxing and inexpensive hobby. The hosting bill for the past year of the blog was around $30. Most of my time is spent working on unrelated efforts which I am not at liberty to make public. Currently, they leave precious-little time and energy for my long-term (think lifetime) project. Feel free, of course, to imagine that I am an idler sitting in an armchair drinking the days away, if it makes you happy.
        troutwine 15y ago
wnoise 15y ago
Eh, in lazily-evaluated languages such Haskell, there is much less need for macros.
- raganwald 15y ago
  True, but can't we say that about every non-trivial PL feature? For example, we could say that: In OOP languages such as Smalltalk, there is much less need for macros. The author's point is that programming languages are great when you're cutting with the grain, when you're working in the domain the author optimized the language around. But when you leave the domain, macros are an important tool. Don't get me wrong, I think that lazy evaluation is an incredibly valuable tool for separating what from how. But I wonder how it obviates the need for macros any more so than a well-defined OO system or any other non-trivial language feature.
  - wnoise 15y ago
    sedachv actually got my point. The standard need for macros rather than functions is to handle staged computations, distinguishing between names and objects. Lazy evaluation really does strike right at the heart of this, unlike something like "object system".
    - kenjackson 15y ago
      Closures give you this (staged computations). Macros really give syntactic abstraction. That's it. That's nice, but you get the rest via other mechanisms.
      - sedachv 15y ago
        Closures can't do partial evaluation, macros can.
        kenjackson 15y ago
        I wasn't trying to imply you can do everything with closures. But rather you could do lazy evaluation with it (I guess staged computations could include various forms of pre-rutime evaluation).
joev 15y ago
Coral cached version: http://www.loper-os.org.nyud.net/?p=401 http://www.loper-os.org.nyud.net/?p=401
primodemus 15y ago
"Wadler created ML to help automate proofs". Minor nitpick: actually, ML was designed by Robin Milner: http://en.wikipedia.org/wiki/ML_(programming_language) http://en.wikipedia.org/wiki/ML_(programming_language)
ekidd 15y ago
I like macros and have used them heavily for most of my career. But in practice, macros have some limitations: 1. Macros allow you to easily improve your syntax, but offer less guidance for improving your semantics. To use a mathematical analogy, it's nice to have a better notation, but what you really want are better definitions and theorems. Most macros are thin syntactic wrappers; few offer profound semantic insights. (Although there are some lovely examples of the latter in PG's excellent On Lisp: http://www.paulgraham.com/onlisp.html http://www.paulgraham.com/onlisp.html .) 2. Macros almost never call macroexpand recursively, except inside of code bodies. This means that macros like defclass usually cannot be extended in an incremental or modular fashion. Instead, you need to declare a def-your-class macro that can't usefully compose with any other def-her-class macro that you might encounter. 3. Weaker mechanisms than macros can accomplish nearly as much, with fewer issues similar to (2). For example, you can re-implement many Lisp macros using Ruby's block syntax, metaobject protocol, and ability to call class methods from inside a class body. And you can usually compose the resulting DSLs freely, even when they're analogous to defclass extensions—witness all the ActiveRecord add-ons. So I like macros, but I no longer think that they're the ultimate high-level abstraction. They're mostly a nice way to incrementally improve your notation, except in the hands of programming language designers who are prepared to pull out all the stops. Vladimir Sedach: The Haskell gang is primarily interested in advancing applied type theory. If we're talking about high-level abstractions, this is probably not the best way to describe Haskell. (In fact, Haskell's type system is notoriously ad hoc, compared to languages like ML.) Originally, Haskell started out a project to explore lazy evaluation and programming without side effects. But in recent years, the Haskell community has been investigating ways to build better domain-specific languages using ideas from mathematics: combinators, monads, co-monads, arrows, derivatives of types, and so on. So far, many of these Haskell ideas have a steep learning curve, but the results are nonetheless impressive. If I were trying to design a "hundred-year language", I would definitely pay close attention to the Haskell community—they just bubble with fascinating ideas. And mathematical ideas tend to endure.
- sedachv 15y ago
  > Macros allow you to easily improve your syntax, but offer less guidance for improving your semantics. This IMO is the biggest problem with macro usage today and needs to be overcome. I've started a campaign against the "macros are syntactic sugar, and I have a sweet tooth!" camp (http://carcaddar.blogspot.com/2010/08/input-needed.html http://carcaddar.blogspot.com/2010/08/input-needed.html). I recommend reading this excellent parody blog post (written at the height of PG-inspired Lisp mania) for background: http://classic-web.archive.org/web/20070706135848/brucio.blogspot.com/2005/03/concision-is-equivalent-to.html http://classic-web.archive.org/web/20070706135848/brucio.blo... > Macros almost never call macroexpand recursively, except inside of code bodies. This is by design. The expansion of a macro needs to be opaque - otherwise where is the abstraction? When you macroexpand, you're opening the black box. You can only rely on macroexpanding macros that you have some control over. > This means that macros like defclass usually cannot be extended in an incremental or modular fashion. This is because it's the same problem as extending an arbitrary grammar. You can't compose two grammars and expect the result to be non-ambiguous. > Instead, you need to declare a def-your-class macro that can't usefully compose with any other def-her-class macro that you might encounter. This is completely, totally wrong. Things like def-your-class is the reason I started my education campaign. You should never write a def-your-class macro. Macros are the wrong way to solve this problem, which is why CLOS and the CLOS metaobject protocol include all the facilities you could want for extending defclass in a composable way (Art of the Metaobject Protocol is a comprehensive, but confusing, reference).
swannodette 15y ago
If you want to understand the beauty of macros don't read silly trollish blogs. I've said it a million times and I'll say it a million more - look at very good Lisp code that wisely unites elegant runtime code with just enough macro sugar to have something beautiful and idiomatic. So far I can think of no better text than The Reasoned Schemer, http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&tid=10663 http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&#38.... You get the soul of Prolog in 200 lines of Scheme R5RS. The entire system fits on 2 pages. This book is written by three true masters of macros and functional programming Oleg Kiselyov, William Byrd, and Daniel P. Friedman.
- silentbicycle 15y ago
  Thank you for suggesting good code to read, so many comments like this just leave people hanging. :) I know great C code to suggest, but draw a blank when it comes to Lisp macros done right.

99 points by MrJagil 15y ago

I told a friend how I was going to implement my next project in Lisp. He immediately thought I was nuts to do so -- that it would never get acquired or it would be too hard to hire people to work on when it grows. All of the tried-and-true objections to using Lisp for a "real" project. Then I read an article like this and realize I've been explaining it all wrong. This is a great read and a real eye-opener.

sedachv 15y ago
Give your programmers a copy of _Practical Common Lisp_, it's not difficult to learn.

kenjackson 15y ago

What do CRUD apps have to do with Lisp macros? The points are disconnected and points are introduced with no evidence or logic to back them. The only thing I can gather is this guy is about to become a billionaire, because he has found a way to rid the world of spreadsheets and CRUD apps... apparently with steel mills.

troutwine 15y ago
I try to think the best of people. With no code to Loper OS that I can find and a writing style that perpetuates lisp elitism using peculiar analogies to the Industrial Revolution without any evidence that this elitism is warrented--I use lisp, I like lisp but it's one tool in a bag--I find it hard to believe that Mr. Datskovskiy is serious. Surely this is brilliant satire?
- asciilifeform 15y ago
  > without any evidence that this elitism is warranted What kind of evidence would you accept? And if you won't accept it from R. Gabriel, P. Graham, P. Greenspun, and other "household" names, why should the rest of us bother sweating to assemble a watertight case for your wastebasket's eyes?
  - troutwine 15y ago
    > What kind of evidence would you accept? Code. Every author takes on himself the onerous to justify his claims. I am familiar with the works of the authors you mention; they present a good argument that lisp is a fine language for fine works. It is an entirely different argument to assert that, merely through the use of lisp, you will revolutionize the world. That, as I read it, is the claim made by the original author. It is that claim which has gone unsubstantiated, for years. The rest of you--incidentally, no need to use exclusionary language: we are all friend here--need do nothing. Mr. Datskovskiy has some code to deliver. A bootloader in two years is no accomplishment, though Mr. Datskovskiy has excelled in punditry in this period.
    - asciilifeform 15y ago
      Original author speaking. If you actually read the blog (unbearable ordeal, I understand) you would know that Lisp (in available incarnations on available hardware) is not enough: http://www.loper-os.org/?p=284 http://www.loper-os.org/?p=284 Let's see how much of an OS and hardware architecture you can design in several years - without using a single line of code written by another person. Starting with the NAND gate - because that is where you will have to start if you truly intend to avoid recapitulating the mistakes of the past three decades. There was no sequel to the bootloader - I have been occupied with reverse-engineering the bitstream format of a major brand of FPGA (no naming names) in order to dispense with the proprietary x86 toolchain (and eventually with the abomination called Verilog.) Publishing incremental progress in bite-sized chunks is vastly over-rated. It wastes everyone's time and contributes to the proliferation of hideously ragtag systems reminiscent of "Junkyard Wars" (aka much of the software industry.) Punditry, on the other hand, is a relaxing and inexpensive hobby. The hosting bill for the past year of the blog was around $30. Most of my time is spent working on unrelated efforts which I am not at liberty to make public. Currently, they leave precious-little time and energy for my long-term (think lifetime) project. Feel free, of course, to imagine that I am an idler sitting in an armchair drinking the days away, if it makes you happy.
      - troutwine 15y ago

wnoise 15y ago

Eh, in lazily-evaluated languages such Haskell, there is much less need for macros.

raganwald 15y ago
True, but can't we say that about every non-trivial PL feature? For example, we could say that: In OOP languages such as Smalltalk, there is much less need for macros. The author's point is that programming languages are great when you're cutting with the grain, when you're working in the domain the author optimized the language around. But when you leave the domain, macros are an important tool. Don't get me wrong, I think that lazy evaluation is an incredibly valuable tool for separating what from how. But I wonder how it obviates the need for macros any more so than a well-defined OO system or any other non-trivial language feature.
- wnoise 15y ago
  sedachv actually got my point. The standard need for macros rather than functions is to handle staged computations, distinguishing between names and objects. Lazy evaluation really does strike right at the heart of this, unlike something like "object system".
  - kenjackson 15y ago
    Closures give you this (staged computations). Macros really give syntactic abstraction. That's it. That's nice, but you get the rest via other mechanisms.
    - sedachv 15y ago
      Closures can't do partial evaluation, macros can.
      - kenjackson 15y ago
        I wasn't trying to imply you can do everything with closures. But rather you could do lazy evaluation with it (I guess staged computations could include various forms of pre-rutime evaluation).

joev 15y ago

Coral cached version: http://www.loper-os.org.nyud.net/?p=401 http://www.loper-os.org.nyud.net/?p=401

primodemus 15y ago

"Wadler created ML to help automate proofs". Minor nitpick: actually, ML was designed by Robin Milner: http://en.wikipedia.org/wiki/ML_(programming_language) http://en.wikipedia.org/wiki/ML_(programming_language)

ekidd 15y ago

I like macros and have used them heavily for most of my career. But in practice, macros have some limitations: 1. Macros allow you to easily improve your syntax, but offer less guidance for improving your semantics. To use a mathematical analogy, it's nice to have a better notation, but what you really want are better definitions and theorems. Most macros are thin syntactic wrappers; few offer profound semantic insights. (Although there are some lovely examples of the latter in PG's excellent On Lisp: http://www.paulgraham.com/onlisp.html http://www.paulgraham.com/onlisp.html .) 2. Macros almost never call macroexpand recursively, except inside of code bodies. This means that macros like defclass usually cannot be extended in an incremental or modular fashion. Instead, you need to declare a def-your-class macro that can't usefully compose with any other def-her-class macro that you might encounter. 3. Weaker mechanisms than macros can accomplish nearly as much, with fewer issues similar to (2). For example, you can re-implement many Lisp macros using Ruby's block syntax, metaobject protocol, and ability to call class methods from inside a class body. And you can usually compose the resulting DSLs freely, even when they're analogous to defclass extensions—witness all the ActiveRecord add-ons. So I like macros, but I no longer think that they're the ultimate high-level abstraction. They're mostly a nice way to incrementally improve your notation, except in the hands of programming language designers who are prepared to pull out all the stops. Vladimir Sedach: The Haskell gang is primarily interested in advancing applied type theory. If we're talking about high-level abstractions, this is probably not the best way to describe Haskell. (In fact, Haskell's type system is notoriously ad hoc, compared to languages like ML.) Originally, Haskell started out a project to explore lazy evaluation and programming without side effects. But in recent years, the Haskell community has been investigating ways to build better domain-specific languages using ideas from mathematics: combinators, monads, co-monads, arrows, derivatives of types, and so on. So far, many of these Haskell ideas have a steep learning curve, but the results are nonetheless impressive. If I were trying to design a "hundred-year language", I would definitely pay close attention to the Haskell community—they just bubble with fascinating ideas. And mathematical ideas tend to endure.

sedachv 15y ago
> Macros allow you to easily improve your syntax, but offer less guidance for improving your semantics. This IMO is the biggest problem with macro usage today and needs to be overcome. I've started a campaign against the "macros are syntactic sugar, and I have a sweet tooth!" camp (http://carcaddar.blogspot.com/2010/08/input-needed.html http://carcaddar.blogspot.com/2010/08/input-needed.html). I recommend reading this excellent parody blog post (written at the height of PG-inspired Lisp mania) for background: http://classic-web.archive.org/web/20070706135848/brucio.blogspot.com/2005/03/concision-is-equivalent-to.html http://classic-web.archive.org/web/20070706135848/brucio.blo... > Macros almost never call macroexpand recursively, except inside of code bodies. This is by design. The expansion of a macro needs to be opaque - otherwise where is the abstraction? When you macroexpand, you're opening the black box. You can only rely on macroexpanding macros that you have some control over. > This means that macros like defclass usually cannot be extended in an incremental or modular fashion. This is because it's the same problem as extending an arbitrary grammar. You can't compose two grammars and expect the result to be non-ambiguous. > Instead, you need to declare a def-your-class macro that can't usefully compose with any other def-her-class macro that you might encounter. This is completely, totally wrong. Things like def-your-class is the reason I started my education campaign. You should never write a def-your-class macro. Macros are the wrong way to solve this problem, which is why CLOS and the CLOS metaobject protocol include all the facilities you could want for extending defclass in a composable way (Art of the Metaobject Protocol is a comprehensive, but confusing, reference).

swannodette 15y ago

If you want to understand the beauty of macros don't read silly trollish blogs. I've said it a million times and I'll say it a million more - look at very good Lisp code that wisely unites elegant runtime code with just enough macro sugar to have something beautiful and idiomatic. So far I can think of no better text than The Reasoned Schemer, http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&tid=10663 http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&#38.... You get the soul of Prolog in 200 lines of Scheme R5RS. The entire system fits on 2 pages. This book is written by three true masters of macros and functional programming Oleg Kiselyov, William Byrd, and Daniel P. Friedman.

silentbicycle 15y ago
Thank you for suggesting good code to read, so many comments like this just leave people hanging. :) I know great C code to suggest, but draw a blank when it comes to Lisp macros done right.