4 ms·

Since many/most Lisp macros implement syntax, there can be a lot of syntax in Lisp, too. The parsing of the syntax is distributed over the macro definitions, so

• kazinator 10y ago
  Macros do not have to deal with syntax at the level of "how does this sequence of tokens reshape into a tree". (Not usually; exceptions are easily contrived and exist in the wild.) Usually, the syntax is already parsed when it comes into a macro. E,g: (sentence-macro subject object verb) sentence-macro doesn't have guess what part of the utterance is the object and which is the verb. The verb is the third argument, and that is that. This is the case whether it is a single word, or a phrase. Though this is still syntax, it is parsed syntax. When we say that Lisp has no syntax, it means that it dosen't have that silly, counterproductive stuff naively imitated from natural languages which represents the tree in a way that requires mind-bending work to recover.
  • taeric 10y ago
    You should probably check out the LOOP macro sometime. Examples include: (loop for i from 0 to 10 for j from 10 downto 0 collect (cons i j)) That is a fully legit expression. So, if you consider LOOP contrived, then you have a point. It is a very useful and a very powerful macro, though. Not sure why it wouldn't count.
    • kazinator 10y ago
      LOOP does speak to what you can do with macros. Obviously, a macro can interpret its arguments however it wants. You can implement a language that requires GLR (or worse). LOOP is also not universally loved in the Lisp world. It's fine if you don't need to extend it. Not only is it inherently inextensible (for no good reason), but it's hostile to wrapping. If you write a MYLOOP macro which passes most of its arguments to LOOP and adds a few clauses of its own (translated to LOOP syntax), you have to parse the LOOP syntax to see where your clauses are and in which order in relation to the standard ones. (Unless you do something hacky or ugly, like recognize your extensions by some delimiters, without regard for the surrounding syntax, so that they don't actually blend in.) LOOP is the only high level control construct I've ever seen anywhere in which you can achieve nonportable behaviors: because of the construct itself, not because of nonportable expressions plugged into it. The treatment of clause symbols as character strings, so that CL:FOR and MYPACKAGE:FOR can both head off a for clause is a design smell. LOOP should have been designed to use symbols in the CL package, so that if someone wants to use APPENDING, they nicely import that symbol or pick it up via :use. LOOP asks you to use different syntax for walking a vector and list, when the rest of Lisp has a sequences library that lets you use common functions for both. Supporting the specialized syntax is fine (just like the library also has list and vector specific things), but the lack of a generic sequence iterating clause shows a discord with the rest of the language. It didn't require much imagination to have, say a FOR x OVER <list-string-or-vec>. We can have a perfectly good Lisp dialect without any sort of mini-language parsing macro like LOOP. It illustrates what you can do with macros, not what is usually done with macros. Macros usually leverage their tree structure and destructuring not to do any parsing work, and focus on the transformation and its semantics.
      • lispm 10y ago
        > Not only is it inherently inextensible (for no good reason) Various LOOP implementations are extensible. > Macros usually leverage their tree structure and destructuring not to do any parsing work, and focus on the transformation and its semantics. Untrue. Many macros need custom destructuring or even walking the code in some way. > We can have a perfectly good Lisp dialect without any sort of mini-language parsing macro like LOOP. Sure: basic Scheme, but then Scheme has its own complex looping constructs like http://wiki.call-cc.org/eggref/4/foof-loop http://wiki.call-cc.org/eggref/4/foof-loop . Iteration macros like LOOP, ITERATE, FOR and others are very convenient and powerful.
        • kazinator 10y ago
          I mostly agree without, except that custom destructuring and code walking do not imply parsing. Destructuring is just how we access the tree (already parsed object). Sometimes the tree isn too complicated for the simple destructuirng performed by destructuring macro lambda lists, so we have to do things like walk substructures and apply DESTRUCTURING-BIND or use some pattern matching library or whatever. Example: just because I I have some MAPCAR over a list of variable-init pairs (qualifies as "custom destructuring") doesn't mean I'm parsing syntax that hasn't been parsed. I'm just destructuring a syntax tree that hasn't been destructured. Destructuring isn't parsing. Destructuring is not even mentioned in papers and textbooks on compilers; it falls into the bucket of somehow walking the tree, which falls under semantic analysis. Code walking is ultimately done to the expansion of every macro. COMPILE and EVAL perform code walking. No textbook on compilers will refer to code walks over an AST as part of the parsing stage. Of course the point is valid that Lisp macros sometimes take a flat list of items and apply recursive phrase structure rules to recover a tree (or small subtree, as the case may be). This is rare, and basically a last resort device; if you're doing that, you're writinig some sort of "big deal" macro. Instances of it are rare in Common Lisp, and I don't suspect it is done frequently in Lisp programs. It's great that wan do that. It's also great that because of the way the langauge works, we can accomplish a lot without having to do that. Speaking of ITERATE, its parsing of clauses is trivial, because it uses Lispy syntax. Dealing with (for var = expr) versus (for var initially expr then expr) versus (for var first expr then expr) is "quasi parsing". OK, we have a FOR. What isn the third item? Switch on a bunch of cases: =, INITIALLY, FIRST, .... If it's unrecognized, then signal an error. In each of these cases, certain things are fixed by position in the syntax already. This is "parsing" only in the sense that Unix people refer to simple trivial line argument processing as "parsing". (Only a few utils do actual parsing or a recursive grammar, examples being find and tcpdump.)
  • lispm 10y ago
    > Macros do not have to deal with syntax at the level of "how does this sequence of tokens reshape into a tree" That's not syntax. Syntax is concerned whether a sequence of words are valid expressions in a language and determines syntactic categories for these. You can lookup a better definition of syntax, I'm too lazy. These are all DEFUN forms. Some are valid Lisp, some are not. (defun foo () ()) is valid (defun () foo ()) is invalid (defun () () foo) is invalid (defun (foo) () foo) is invalid (defun foo () foo) is valid (defun () foo foo) is invalid (defun foo foo ()) is invalid (defun (setf foo) (setf foo) foo) is valid (defun foo (setf foo) (setf foo)) is invalid (defun (setf foo) foo (setf foo)) is invalid Just by changing the order of subforms in a macro form we can produce valid Lisp and invalid Lisp forms. (defmacro defun (name args &body forms) ...) Does not tell you that. You need to implement that logic in the macro somewhere. > sentence-macro doesn't have guess what part of the utterance is the object and which is the verb. The verb is the third argument, and that is that. Macros implement more complex syntax. Check the ANSI CL specification and its EBNF syntax declarations some time. > When we say that Lisp has no syntax Then it's just wrong and misleading. Take the DEFUN macro: the EBNF (extended backus naur form) syntax definition for DEFUN is: defun function-name lambda-list [[declaration* | documentation]] form* Is (defun (bla blub) (&rest foo &optional bar) (declare fixnum) "oops" ((((fooo)))))) a valid defun expression ???????????????? The macro has to check that. It better rejects invalid expressions. It also has to look at the elements of the form to destructure them in the right way, so that it can process them and create a new valid form. The Lisp implementation provides for the implementation of DEFUN as much as: (defmacro defun (spec args &body body ...) ...) The macro language does not allow further specifications of the name the arglist or the body in the macro interface. All it gets are spec, args and body. Now the macro has to implement the syntax for spec, args and body. Questions the macro has to answer: * is the function name a symbol or a list of the form (setf foo) * is the arglist a valid lambda-list? Now check the EBNF syntax for lambda lists with whole/optional/key/rest options with default values and what have you. * now it has to parse the body: * is the declaration valid? Now check the EBNF syntax for declarations to see what needs to be done. * is the documentation a string at the right position? * is the body a sequence of forms? Now check the EBNF syntax for FORM. This all has to be backed into the DEFUN macro somehow or checked from there. And not all implementations are good at it. Various syntax errors: The name is not valid: CL-USER 14 > (defun (foo bar) baz (list)) Error: (FOO BAR) is neither of type SYMBOL nor a list of the form (SETF SYMBOL). 1 (abort) Return to level 0. 2 Return to top loop level 0. A symbol is not a valid lambda list: CL-USER 16 > (defun (setf bar) baz (list)) (SETF BAR) CL-USER 17 > (compile '(setf bar)) Error: Invalid lambda list: BAZ Keyword argument is wrong: CL-USER 19 > (defun foo (&key ((aa))) (list)) FOO CL-USER 20 > (compile 'foo) Error: Malformed (keyword variable) form in &key argument ((AA)) Wrong declaration: CL-USER 24 > (defun foo (&key (aa)) (declare inline-function foo)) Error: Alist element INLINE-FUNCTION is not a cons or NIL Wrong form: CL-USER 26 > (defun foo (&key (aa)) (declare (inline foo)) (((lambda () (lamda () ()))))) FOO CL-USER 27 > (compile 'foo) Error: Illegal car ((LAMBDA NIL (LAMDA NIL NIL))) in compound form (((LAMBDA NIL #))). And so on. An INFIX macro: (infix 3 + 2 ^ 5) It has to implement infix syntax. If it is still not clear, below is the syntax for the LOOP macro. The LOOP implementation has to implement the syntax, so that (loop for i below 70 do (print i)) is recognized as a valid program and it better detect that (loop do (print i) for i below 70 ) is not a valid program, because it violates the syntax below. The ``simple'' loop form: loop compound-form* => result* The ``extended'' loop form: loop [name-clause] {variable-clause}* {main-clause}* => result* name-clause::= named name variable-clause::= with-clause | initial-final | for-as-clause with-clause::= with var1 [type-spec] [= form1] {and var2 [type-spec] [= form2]}* main-clause::= unconditional | accumulation | conditional | termination-test | initial-final initial-final::= initially compound-form+ | finally compound-form+ unconditional::= {do | doing} compound-form+ | return {form | it} accumulation::= list-accumulation | numeric-accumulation list-accumulation::= {collect | collecting | append | appending | nconc | nconcing} {form | it} [into simple-var] numeric-accumulation::= {count | counting | sum | summing | } maximize | maximizing | minimize | minimizing {form | it} [into simple-var] [type-spec] conditional::= {if | when | unless} form selectable-clause {and selectable-clause}* [else selectable-clause {and selectable-clause}*] [end] selectable-clause::= unconditional | accumulation | conditional termination-test::= while form | until form | repeat form | always form | never form | thereis form for-as-clause::= {for | as} for-as-subclause {and for-as-subclause}* for-as-subclause::= for-as-arithmetic | for-as-in-list | for-as-on-list | for-as-equals-then | for-as-across | for-as-hash | for-as-package for-as-arithmetic::= var [type-spec] for-as-arithmetic-subclause for-as-arithmetic-subclause::= arithmetic-up | arithmetic-downto | arithmetic-downfrom arithmetic-up::= [[{from | upfrom} form1 | {to | upto | below} form2 | by form3]]+ arithmetic-downto::= [[{{from form1}}1 | {{{downto | above} form2}}1 | by form3]] arithmetic-downfrom::= [[{{downfrom form1}}1 | {to | downto | above} form2 | by form3]] for-as-in-list::= var [type-spec] in form1 [by step-fun] for-as-on-list::= var [type-spec] on form1 [by step-fun] for-as-equals-then::= var [type-spec] = form1 [then form2] for-as-across::= var [type-spec] across vector for-as-hash::= var [type-spec] being {each | the} {{hash-key | hash-keys} {in | of} hash-table [using (hash-value other-var)] | {hash-value | hash-values} {in | of} hash-table [using (hash-key other-var)]} for-as-package::= var [type-spec] being {each | the} {symbol | symbols | present-symbol | present-symbols | external-symbol | external-symbols} [{in | of} package] type-spec::= simple-type-spec | destructured-type-spec simple-type-spec::= fixnum | float | t | nil destructured-type-spec::= of-type d-type-spec d-type-spec::= type-specifier | (d-type-spec . d-type-spec) var::= d-var-spec var1::= d-var-spec var2::= d-var-spec other-var::= d-var-spec d-var-spec::= simple-var | nil | (d-var-spec . d-var-spec) Arguments and Values: compound-form---a compound form. name---a symbol. simple-var---a symbol (a variable name). form, form1, form2, form3---a form. step-fun---a form that evaluates to a function of one argument. vector---a form that evaluates to a vector. hash-table---a form that evaluates to a hash table. package---a form that evaluates to a package designator. type-specifier---a type specifier. This might be either an atomic type specifier or a compound type specifier, which introduces some additional complications to proper parsing in the face of destructuring; for further information, see Section 6.1.1.7 (Destructuring). result---an object.