3 ms·

> So your point that the Lisp evaluator doesn't even understand sequences of characters is true for any compiler, they all operate on the AST. Lisp does not re

by lispm 5y ago

> So your point that the Lisp evaluator doesn't even understand sequences of characters is true for any compiler, they all operate on the AST. Lisp does not really operate on an AST. It operates on nested token lists, without syntax representation. For example (postfix 1 2 +) can be legal in Lisp, because it does not (!) parse that code according to a syntax before handing it to the evaluator. Lisp code consists of nested lists of data. Textual Lisp code uses a data format for these nested lists, which can be read and printed. A lot of Lisp code, though, is generated without being read/printed -> via macros.

chaosite 5y ago
You're arguing semantics, I think. I would simply say that Lisp's AST is S-expressions (those nested token lists), and that the parser is Lisp's read function. Then your example is just something that's allowed by Lisp's syntax, while something like ')postfix 1 2 +(' would be something that's not allowed by the syntax. What you say about Lisp code being generated without being read or printed is of course true, and while Lisp takes that idea and runs with it, it's not exactly unique to Lisp either; Rust's macro can do the same thing, without S-expressions. In other languages you usually generate source code, for example Java has a lot of source code generators (e.g., JAXB's XJC that used to come with the JDK).
- lispm 5y ago
  The parser for s-expressions is READ. S-Expressions are just a data syntax and know nothing about the Lisp programming language syntax. Lisp syntax is defined on top of s-expressions. The Lisp evaluator sees no parentheses and no text, It would not care if the input text contains )postfix 1 2+( . The reader can actually be programmed to accept that as input. The actual Lisp forms need to be syntax checked then by a compiler or interpreter. There are lots of language with code generators. Lisp does this as core part of the language and can do it at runtime as part of the evaluation machinery.
kazinator 5y ago
If (postfix 1 2+) is ready to be handed to the evaluator, it's because it has been parsed. That means it must be a parsed representation. "Parse tree" doesn't apply because parse trees record token-level details; ( and ) are tokens, yet don't appear to the evaluator. "Abstract syntax tree" is better, though doesn't meet some people's expectations if they have worked on compilers that had rich AST nodes with lots of semantic properties. The constitutents of the list are not "tokens" in Common Lisp. ANSI CL makes it clear that the characters "postfix", in the default read table, are token constituents; they get gathered into a token until the space appears. That token is then converted into a symbol name, which is interned to produce a symbol. That symbol is no longer a "token".

by lispm 5y ago

chaosite 5y ago

You're arguing semantics, I think. I would simply say that Lisp's AST is S-expressions (those nested token lists), and that the parser is Lisp's read function. Then your example is just something that's allowed by Lisp's syntax, while something like ')postfix 1 2 +(' would be something that's not allowed by the syntax. What you say about Lisp code being generated without being read or printed is of course true, and while Lisp takes that idea and runs with it, it's not exactly unique to Lisp either; Rust's macro can do the same thing, without S-expressions. In other languages you usually generate source code, for example Java has a lot of source code generators (e.g., JAXB's XJC that used to come with the JDK).

lispm 5y ago
The parser for s-expressions is READ. S-Expressions are just a data syntax and know nothing about the Lisp programming language syntax. Lisp syntax is defined on top of s-expressions. The Lisp evaluator sees no parentheses and no text, It would not care if the input text contains )postfix 1 2+( . The reader can actually be programmed to accept that as input. The actual Lisp forms need to be syntax checked then by a compiler or interpreter. There are lots of language with code generators. Lisp does this as core part of the language and can do it at runtime as part of the evaluation machinery.

kazinator 5y ago

If (postfix 1 2+) is ready to be handed to the evaluator, it's because it has been parsed. That means it must be a parsed representation. "Parse tree" doesn't apply because parse trees record token-level details; ( and ) are tokens, yet don't appear to the evaluator. "Abstract syntax tree" is better, though doesn't meet some people's expectations if they have worked on compilers that had rich AST nodes with lots of semantic properties. The constitutents of the list are not "tokens" in Common Lisp. ANSI CL makes it clear that the characters "postfix", in the default read table, are token constituents; they get gathered into a token until the space appears. That token is then converted into a symbol name, which is interned to produce a symbol. That symbol is no longer a "token".