4 ms·

> It was the reader which settled all the syntactic facts about what operation is applied to what operands. The reader does not know anything about operands or

by lispm 4y ago

> It was the reader which settled all the syntactic facts about what operation is applied to what operands. The reader does not know anything about operands or operators. It does not even know if the thing read is data or code. It also does not know if it is valid code. Enter (3 4 *) and the reader happily takes that expression, not caring about operators and operands. The reader will print zero warning about such expressions. To really know what the symbols in an expression actually are, one would need eval, compile or a code walker. Are they used as literal symbols? variables? functions? macros? block names? type names? ... > Because, look: (- 10) and (- 10 1) behave differently: the 10 operand changes semantic role between minuend and subtrahend based on syntax. All that eval knows is that it's passing arguments. That's semantics, not syntax. EVAL and COMPILE check that it is a function call, what arglist it expects (zero or more args, &optional args, &rest args, keyword args, and process them recursively. But Lisp has also special operators, macro operators and a lambda expression. There the syntax requirements are different or even context depended. Also if Lisp reads the expression (foo * +), the reader does not know if FOO is a function or a macro, it thus does not know what * and + are. Are they variables? Are they operators? Something else? They don't even need to have a definition at read time.

kazinator 4y ago
The parser for a C grammar (e.g. Yacc job) also doesn't know anything. Is the output used for pretty-printing (the code is just data), completion, or generating code? The reader doesn't know about opperands or operators, but it certainly knows a symbol from an integer or string. We've cleverly arranged things so that it doesn't have to know anything else in order to produce output that, in some cases, can be evaluated directly. > That's semantics, not syntax Syntax versus semantics is sometimes about where you put the goalposts. What if I make it a macro that looks at the argument count and chooses a different function? (defmacro minus (&rest args) (if (eql 1 (length args)) `(unary-minus ,(car args)) ;; takes exactly one arg `(nary-minus ,@args))) ;; blows up if not given at least two args Is that still "semantics, not syntax"? Why does it become semantics if one function does it itself by switching on the argument count internally? Everything is syntax! Adding two floating-point value is syntax: the representation has to be parsed into sign bit, exponent and mantissa, and so on before anything happens. Sure, those fields are accessed conveniently by position, but so are the elements of the list (sign exponent mantissa), and our goalposts say that that is syntax. The parsing never stops. The CPU fetches instructions, decodes (== parsing!) and executes. When the Lisp run-time looks at a value to extract the tag bits, and then further delves into the object accordingly, it is parsing syntax.
- lispm 4y ago
  > The reader doesn't know about opperands or operators, but it certainly knows a symbol from an integer or string. That's the s-expression syntax. Beyond that, symbols have no special meaning for the reader. > Everything is syntax Syntax describes the structure of a language. Wikipedia: "the syntax of a computer language is the rules that defines the combinations of symbols that are considered to be correctly structured statements or expressions in that language." CLHS 1.4.1.2 "This specification uses an extended Backus Normal Form (BNF) to describe the syntax of Common Lisp macro forms and special forms." So the ANSI CL standard claims that its macro forms and special forms have syntax and that this syntax is described by an eBNF? Example: DO do ({var | (var [init-form [step-form]])}*) (end-test-form result-form*) declaration* {tag | statement}* If I try this in SBCL * (do "hello") debugger invoked on a SB-KERNEL::ARG-COUNT-ERROR in thread #<THREAD "main thread" RUNNING {1004AC01C3}>: Error while parsing arguments to DEFMACRO DO: too few elements in ("hello") to satisfy lambda list (VARLIST ENDLIST &BODY BODY): at least 2 expected, but got 1 It tells me that there is an error while parsing (!) arguments to the DEFMACRO DO. Woops? There is a parser? There is an error detected? So, beyond the the reader SBCL seems to claim it has a parser?
  - kazinator 4y ago
    You used the argument that the Lisp reader isn't parsing Lisp because it doesn't actually know Lisp; it has no idea what the operators mean. But, above, the message "error while parsing arguments to DEFMACRO DO" is just a destructuring error on "macro lambda list" pattern match on a piece of datum: the unevaluated macro call. That pattern matcher has no idea what DO means, in much same way that the reader doesn't know what DO means. The reader has done so much work, that the remaining parsing is just a very easy job which enjoys easy positional access to everything in the tree, at every level, at worst having to loop over a sequence of similar items. The syntax notation like: do ({var | (var [init-form [step-form]])}*) (end-test-form result-form*) declaration* {tag | statement}* is relying on read syntax; it has parentheses in it! Moreover, it is a greatly condensed specification, due to its reliance on the properties of the read syntax. The init-form can be anything: a struct literal, vector, string, character. Yet, I don't see the rules for that in this grammar; they given elsewhere, and just assumed here. If everything were to be made explicit here, the way a handful of parentheses have been made explicit, it would be pretty large specification, of which the do-specific parts above would be small. The do-specific material is just a handful of self-contained patterns applied to individual objects that have already been parsed out by the reader. For instance, the pattern pattern {var | (var [init-form [step-form]])}* is understood to be applied to that part of the form which can be conveniently retrieved as (cadr form). The amount of work left is very small, of low complexity. The pattern is not a recursive language; it is a regular language (if we pretend that var, init-form and step-form are terminal symbols). When the (var init-form step-form) is matched against the corresponding object, the step-form is easily retrieved by position, regardless of the complexity of init-form. This is not true of the unparsed expression in which scanning across an init-form to get to step-form is a complex step. The reader has done the work which allows this stage of processing to treat "init-form" as if it were a terminal symbol in a simple regular grammar that is just concerned with a positionally-determined portion of the do syntax. It looks like Lisp has a lot of syntax beyond the read syntax if we look at all of the forms together. They all share the read syntax, but have their unique bits that drive their shape. The more of them there are, the larger is the proportion of the language syntax that is contributed by those unique bits. Each time we write a macro and document it, the read syntax doesn't grow, but the amount of deep syntax does. From the perspective of any one S-expression, though, the bulk of the specification of its syntax is in the domain of the reader. The pattern match for a specific shape like do is a small amount of easily digestible additional information.

by lispm 4y ago

kazinator 4y ago

The parser for a C grammar (e.g. Yacc job) also doesn't know anything. Is the output used for pretty-printing (the code is just data), completion, or generating code? The reader doesn't know about opperands or operators, but it certainly knows a symbol from an integer or string. We've cleverly arranged things so that it doesn't have to know anything else in order to produce output that, in some cases, can be evaluated directly. > That's semantics, not syntax Syntax versus semantics is sometimes about where you put the goalposts. What if I make it a macro that looks at the argument count and chooses a different function? (defmacro minus (&rest args) (if (eql 1 (length args)) `(unary-minus ,(car args)) ;; takes exactly one arg `(nary-minus ,@args))) ;; blows up if not given at least two args Is that still "semantics, not syntax"? Why does it become semantics if one function does it itself by switching on the argument count internally? Everything is syntax! Adding two floating-point value is syntax: the representation has to be parsed into sign bit, exponent and mantissa, and so on before anything happens. Sure, those fields are accessed conveniently by position, but so are the elements of the list (sign exponent mantissa), and our goalposts say that that is syntax. The parsing never stops. The CPU fetches instructions, decodes (== parsing!) and executes. When the Lisp run-time looks at a value to extract the tag bits, and then further delves into the object accordingly, it is parsing syntax.

lispm 4y ago
> The reader doesn't know about opperands or operators, but it certainly knows a symbol from an integer or string. That's the s-expression syntax. Beyond that, symbols have no special meaning for the reader. > Everything is syntax Syntax describes the structure of a language. Wikipedia: "the syntax of a computer language is the rules that defines the combinations of symbols that are considered to be correctly structured statements or expressions in that language." CLHS 1.4.1.2 "This specification uses an extended Backus Normal Form (BNF) to describe the syntax of Common Lisp macro forms and special forms." So the ANSI CL standard claims that its macro forms and special forms have syntax and that this syntax is described by an eBNF? Example: DO do ({var | (var [init-form [step-form]])}*) (end-test-form result-form*) declaration* {tag | statement}* If I try this in SBCL * (do "hello") debugger invoked on a SB-KERNEL::ARG-COUNT-ERROR in thread #<THREAD "main thread" RUNNING {1004AC01C3}>: Error while parsing arguments to DEFMACRO DO: too few elements in ("hello") to satisfy lambda list (VARLIST ENDLIST &BODY BODY): at least 2 expected, but got 1 It tells me that there is an error while parsing (!) arguments to the DEFMACRO DO. Woops? There is a parser? There is an error detected? So, beyond the the reader SBCL seems to claim it has a parser?
- kazinator 4y ago
  You used the argument that the Lisp reader isn't parsing Lisp because it doesn't actually know Lisp; it has no idea what the operators mean. But, above, the message "error while parsing arguments to DEFMACRO DO" is just a destructuring error on "macro lambda list" pattern match on a piece of datum: the unevaluated macro call. That pattern matcher has no idea what DO means, in much same way that the reader doesn't know what DO means. The reader has done so much work, that the remaining parsing is just a very easy job which enjoys easy positional access to everything in the tree, at every level, at worst having to loop over a sequence of similar items. The syntax notation like: do ({var | (var [init-form [step-form]])}*) (end-test-form result-form*) declaration* {tag | statement}* is relying on read syntax; it has parentheses in it! Moreover, it is a greatly condensed specification, due to its reliance on the properties of the read syntax. The init-form can be anything: a struct literal, vector, string, character. Yet, I don't see the rules for that in this grammar; they given elsewhere, and just assumed here. If everything were to be made explicit here, the way a handful of parentheses have been made explicit, it would be pretty large specification, of which the do-specific parts above would be small. The do-specific material is just a handful of self-contained patterns applied to individual objects that have already been parsed out by the reader. For instance, the pattern pattern {var | (var [init-form [step-form]])}* is understood to be applied to that part of the form which can be conveniently retrieved as (cadr form). The amount of work left is very small, of low complexity. The pattern is not a recursive language; it is a regular language (if we pretend that var, init-form and step-form are terminal symbols). When the (var init-form step-form) is matched against the corresponding object, the step-form is easily retrieved by position, regardless of the complexity of init-form. This is not true of the unparsed expression in which scanning across an init-form to get to step-form is a complex step. The reader has done the work which allows this stage of processing to treat "init-form" as if it were a terminal symbol in a simple regular grammar that is just concerned with a positionally-determined portion of the do syntax. It looks like Lisp has a lot of syntax beyond the read syntax if we look at all of the forms together. They all share the read syntax, but have their unique bits that drive their shape. The more of them there are, the larger is the proportion of the language syntax that is contributed by those unique bits. Each time we write a macro and document it, the read syntax doesn't grow, but the amount of deep syntax does. From the perspective of any one S-expression, though, the bulk of the specification of its syntax is in the domain of the reader. The pattern match for a specific shape like do is a small amount of easily digestible additional information.