4 ms·

As someone trying to understand what people find special about Lisp, this was helpful. I get it now about macros, but doesn't modifying the language in the comp

As someone trying to understand what people find special about Lisp, this was helpful. I get it now about macros, but doesn't modifying the language in the compiler phase just complicate solving the actual runtime problem? I feel I would get distracted into writing my own DSL. I've never wanted to modify the language as I was writing in it. Now I'm curious.

reikonomusha 4y ago
Don't think of it as modifying the language. Think of it as adding new ways of expressing ideas in the existing language. These sorts of languages/language extensions pop up all the time in "ordinary" programming. For example, common DSLs we see often. - regex is a language for string patterns - JSON is a language for simple nested data - SQL is a language for expressing DB queries - OpenMP contains (essentially) a language for parallelizing loops - (this might sound crazy because it's so common and basically built in standard in almost all programming languages:) infix notation for math The problem is that many of these are absolute kludges in existing languages. Anybody who has had the displeasure of trying to interpolate DB table names into SQL SELECT strings knows this. Most languages try to make it a bit better by adding preprocessors, using builder-patterns, abusing reflection, or otherwise. Looking under the hood of some languages' implementations of these ideas will often lead you to see the most cursed of programmer things in existence. Each of those languages (or close syntactic cousins of them) can be written directly in Lisp, as a library, without shoving them into strings, or using external tools, or making custom file formats. They call can exist as a part of the Lisp language as a syntax extension. Making DSLs requires exercising that muscle a bit; it's non-trivial to design a new and useful language that fits orthogonally into an existing system. It's like being introduced to OO programming where, at least for me, it made sense but it wasn't immediately obvious what a good object model for a problem might be. It took me lots and trial and error to understand how OO works in "the real world", where it's good, and where it falls short. Once you get they hang of making DSLs—or even just small but powerful syntax extensions—it's an amazing tool to have at your disposal.
- michaelwww 4y ago
  My second thought, which I excluded for brevity, was aren't functions and libraries language extensions? I mean technically you could write C without malloc, but it wouldn't be very useful. Regex is a great example of an embedded DSL, but I've never thought "I wish I could write a macro to change it's behavior on the fly in this special case". The last time I looked at this issue I saw that sure, if you don't have generics, it would be helpful to have a language that could write typed functions for you. That makes sense. Changing the parser and compiler behavior before any runtime code executes could be very interesting. There are certainly enough people waxing poetic about the expressiveness of it to make it worth learning.
  - reikonomusha 4y ago
    Again, you don't want to change regex, you want to create it in the first place. Maybe a simpler example, what if I wanted this in (pseudo-)C? with(s = fopen("/foo/bar", ...)) { ...; ...; println(readln(s)); ...; ...; } When the "with" clause ends, the file will automatically be closed. This is more or less bog standard RAII expressed as a new language construct. In Lisp, this is a feature you could build and provide as a library feature. In fact, we can do it, straight here in an HN comment. (defgeneric destroy (x) (:documentation "A destructor for use with the WITH feature".)) (defmacro with ((var = val) &body body) "Python-style RAII macro. VAR will be bound to the value of VAL, and upon exiting scope, will be destroyed with DESTROY." (assert (eq = ':=)) (let ((orig (gensym))) `(let* ((,orig ,val) (,var ,orig)) (unwind-protect (progn ,@body) (destroy ,orig)))) We can register a new destructor easily for open file streams: (defmethod destroy ((s stream)) (close stream)) (We could add some more error handling, etc. to make it more robust.) Now, we effectively have RAII in Lisp: (with (s := (open "/foo/bar" ...)) ... ... (write-line (read-line s)) ... ...) This is now real Lisp code you can actually run, completely integrated into the language, and nicely orthogonal to the existing gamut of features. This is a concept we can use almost immediately after learning about it, instead of waiting for Python's PEP 6363 to pass with consensus from people who may have different programming goals than you do.
srcreigh 4y ago
I don't know how this exactly fits, but you can do crazy stuff like query a SQL db in the compiler to check your Racket program for auto generated variable names. This extends the editor tooling to highlight names which don't match to columns in the DB tables. I wrote a post about this here[1] [1]: http://tech.perpetua.io/2022/01/generating-sqlite-bindings-with-racket-macros/ http://tech.perpetua.io/2022/01/generating-sqlite-bindings-w...

reikonomusha 4y ago

Don't think of it as modifying the language. Think of it as adding new ways of expressing ideas in the existing language. These sorts of languages/language extensions pop up all the time in "ordinary" programming. For example, common DSLs we see often. - regex is a language for string patterns - JSON is a language for simple nested data - SQL is a language for expressing DB queries - OpenMP contains (essentially) a language for parallelizing loops - (this might sound crazy because it's so common and basically built in standard in almost all programming languages:) infix notation for math The problem is that many of these are absolute kludges in existing languages. Anybody who has had the displeasure of trying to interpolate DB table names into SQL SELECT strings knows this. Most languages try to make it a bit better by adding preprocessors, using builder-patterns, abusing reflection, or otherwise. Looking under the hood of some languages' implementations of these ideas will often lead you to see the most cursed of programmer things in existence. Each of those languages (or close syntactic cousins of them) can be written directly in Lisp, as a library, without shoving them into strings, or using external tools, or making custom file formats. They call can exist as a part of the Lisp language as a syntax extension. Making DSLs requires exercising that muscle a bit; it's non-trivial to design a new and useful language that fits orthogonally into an existing system. It's like being introduced to OO programming where, at least for me, it made sense but it wasn't immediately obvious what a good object model for a problem might be. It took me lots and trial and error to understand how OO works in "the real world", where it's good, and where it falls short. Once you get they hang of making DSLs—or even just small but powerful syntax extensions—it's an amazing tool to have at your disposal.

michaelwww 4y ago
My second thought, which I excluded for brevity, was aren't functions and libraries language extensions? I mean technically you could write C without malloc, but it wouldn't be very useful. Regex is a great example of an embedded DSL, but I've never thought "I wish I could write a macro to change it's behavior on the fly in this special case". The last time I looked at this issue I saw that sure, if you don't have generics, it would be helpful to have a language that could write typed functions for you. That makes sense. Changing the parser and compiler behavior before any runtime code executes could be very interesting. There are certainly enough people waxing poetic about the expressiveness of it to make it worth learning.
- reikonomusha 4y ago
  Again, you don't want to change regex, you want to create it in the first place. Maybe a simpler example, what if I wanted this in (pseudo-)C? with(s = fopen("/foo/bar", ...)) { ...; ...; println(readln(s)); ...; ...; } When the "with" clause ends, the file will automatically be closed. This is more or less bog standard RAII expressed as a new language construct. In Lisp, this is a feature you could build and provide as a library feature. In fact, we can do it, straight here in an HN comment. (defgeneric destroy (x) (:documentation "A destructor for use with the WITH feature".)) (defmacro with ((var = val) &body body) "Python-style RAII macro. VAR will be bound to the value of VAL, and upon exiting scope, will be destroyed with DESTROY." (assert (eq = ':=)) (let ((orig (gensym))) `(let* ((,orig ,val) (,var ,orig)) (unwind-protect (progn ,@body) (destroy ,orig)))) We can register a new destructor easily for open file streams: (defmethod destroy ((s stream)) (close stream)) (We could add some more error handling, etc. to make it more robust.) Now, we effectively have RAII in Lisp: (with (s := (open "/foo/bar" ...)) ... ... (write-line (read-line s)) ... ...) This is now real Lisp code you can actually run, completely integrated into the language, and nicely orthogonal to the existing gamut of features. This is a concept we can use almost immediately after learning about it, instead of waiting for Python's PEP 6363 to pass with consensus from people who may have different programming goals than you do.

srcreigh 4y ago

I don't know how this exactly fits, but you can do crazy stuff like query a SQL db in the compiler to check your Racket program for auto generated variable names. This extends the editor tooling to highlight names which don't match to columns in the DB tables. I wrote a post about this here[1] [1]: http://tech.perpetua.io/2022/01/generating-sqlite-bindings-with-racket-macros/ http://tech.perpetua.io/2022/01/generating-sqlite-bindings-w...