5 ms·

The distinction is that Common Lisp source code is not text; it's Lisp values. What you see in a text file of "Lisp source code" isn't actually Lisp source code

The distinction is that Common Lisp source code is not text; it's Lisp values. What you see in a text file of "Lisp source code" isn't actually Lisp source code; it's a text serialization of Lisp source code. Other serializations are possible. As a practical matter, most of the time the distinction doesn't matter. You edit text files. Your Lisp reads the text files and the first thing it does with the contents is convert them to Lisp source code. The source code is then processed as you would expect. One place where the distinction matters is in macro processing: Lisp macro expansion operates on Lisp source code, not on text. If it operated on text, as, for example, C preprocessors do, then macro expansion would be a matter of string substitution. Instead, Lisp macro expanders operate on abstract syntax trees represented as convenient Lisp data structures, which means that macros are not limited to what can be conveniently done with simple string substitution. You have the full Lisp language available, operating on a convenient Lisp representation, to compute the expansion. More generally, when source code is represented as convenient data structures in the language, it's easier to build tools that operate on it. C compilers do not operate directly on text input; they convert it to internal data structures more convenient for the various stages of the compiler to work with. So does Lisp. The difference is that in a C compiler the representations used by the compiler are private implementation details, and in Lisp they are standard surface features of the language, available to every user.

rjsw 9y ago
I would want a Lisp VCS to preserve comments, which are not usually present in the deserialized form of "Lisp source code", so I don't really think your description helps here. I haven't used Xerox Lisp enough to remember whether the Common Lisp side did keep comments in a similar way to InterLisp-D.
- mikelevins 9y ago
  There's no realistic danger of a Common Lisp implementation abandoning support for the text serialization of Lisp source code, because the reader is part of the language spec. That being the case, you can always count on support for Lisp text files, and therefore for comments stored in them. On the other hand, nothing in the standard forbids an implementation from keeping comments around in the image. Off the top of my head I don't know of an implementation that does it, but it might be a handy feature to have.
  - lispm 9y ago
    Comments in Interlisp weren't really magic. Basically the comment function * was taking unevaluated args as a list. An NLAMBDA. It just returned its arg. (* this is a comment in Interlisp) This means also comments could also only be used where they fit into the Lisp syntax and not changed the control flow. In Common Lisp: (defun comment (words) words) (defmacro § (&rest words) `(comment '(,@words))) (let ((a 20)) (cond ((< 0 a 10) (§ Only when A is between 0 and 10. This is a typical value.) (* a 0.1)) (t (§ a is not between 0 and 10) (* a 0.2)))) Now it would need a special pretty printer command, so that * comment lines would be layouted such that they are aligned fit on a line. * the position of the comment should stand out from the code. Interlisp had a pretty printer macro for it. It might be better to turn that into a reader macro, so that inside the comment one would then not be constrained by s-expression syntax.... Interlisp has also the feature to store/retrieve the comment text on demand from a file. Thus the comment text might not be needed to be present in the image itself.
    - nibbula 9y ago
      I'm quite thankful that we don't have to worry about the syntactic position of comments, and the wise folks of CL, as you may know, made it so that saving comment text can be as easy as: (defparameter *comments* nil) (defun comment-saver (stream char) (declare (ignore char)) (push (list *load-pathname* (stream-file-position stream) (with-output-to-string (str) (loop :with c :while (char/= #\newline (setf c (read-char stream))) :do (write-char c str)))) *comments*) (values)) (set-macro-character #\; #'comment-saver) With apologies for boring the non-Lisp infatuated.
      - mikelevins 9y ago
        I think that after 30 years of lively interaction, my relationship with Lisp can be considered to have gone beyond infatuation. Whether that's a good thing is debatable, of course.
skybrian 9y ago
For JavaScript, the Esprima parser [1] converts JavaScript to what appears to be a JSON format. It's formalized by the Estree spec [2]. You can try it out here [3]. JSON is just regular JavaScript data structures (maps and lists). If you want type safety, there are Typescript definitions, but you could ignore that. Another example: Go has a parser and AST that comes with the standard library, which again is just using regular Go data structures (structs and interfaces). If you wanted to write a macro preprocessor, using the built-in parser seems like a much better idea than string manipulation. So, I'm wondering if there's anything more to the way Common Lisp does it? [1] http://esprima.org/ http://esprima.org/ [2] https://github.com/estree/estree https://github.com/estree/estree [3] http://esprima.org/demo/parse.html http://esprima.org/demo/parse.html
- lispm 9y ago
  Lisp does not use a tokenizer or a parser. It uses a reader, which reads a textual representation of data into data objects. A reader is similar to a tokenizer, but the reader just creates an internal representation of some data from an external representation. Basically what a primitive JSON reader would do. A parser would know the programming language syntax and create an internal representation of the program. The Lisp reader does not know anything about the programming language syntax. It just knows about numbers, strings, lists, arrays. But the reader does not know about conditionals, variable declarations, iteration statements, assignment statements, ... var answer = 6 * 7; (defparameter answer (* 6 7)) If we quote it, it is just Lisp data: CL-USER 8 > '(defparameter answer (* 6 7)) (DEFPARAMETER ANSWER (* 6 7)) As you can see the evaluator just returns the data, though the symbols are upcased by default. The data is not enriched as a tokenizer or parser might do. Ideally the data structure prints as it is read. There are no type or syntax annotations necessary/used. Thus it is neither a tokenizer output nor a program parse tree. In your first link the tokenizer breaks up the input in strings and categorizes it. The 42 is read into a string '42' and it is annotated as being a numeric. The Lisp reader does not do that. It reads the two characters 42 and returns an integer number object with the value 42. We can describe this data, it is a list with three elements: CL-USER 9 > (describe *) ; describe the last value (DEFPARAMETER ANSWER (* 6 7)) is a LIST 0 DEFPARAMETER ; a symbol 1 ANSWER ; another symbol 2 (* 6 7) ; a list On this level of s-expressions we know nothing about Lisp as a programming language. It is only known that DEFPARAMETER is a symbol, but not known that it is a language construct and which. It's just a symbol in the first position of a list. Since this list is also a valid Lisp program (because we wrote it as such), we can compute its value and some side-effects. This process is called evaluation. CL-USER 10 > (eval **) ; eval the second last value ANSWER ; it just returns the name of the defined variable We can also input it directly, since the READ-EVAL-PRINT-LOOP already evaluates: CL-USER 11 > (defparameter answer (* 6 7)) ANSWER As you can see, the input to the evaluator is not a tokenizer output and not a parse tree. It's just an expression tree, where the leaves are data objects. It's not (token 3 :type number) or (literal-data :type number :value 3) or something similar. It is just the 3 as a data object. The evaluator itself could be an interpreter or a compiler. The interpreter runs walks the expression tree and computes the values from the expressions, which it sees in data format. A compiler-based evaluator would take the whole expression tree, compile it and then run it.

rjsw 9y ago

I would want a Lisp VCS to preserve comments, which are not usually present in the deserialized form of "Lisp source code", so I don't really think your description helps here. I haven't used Xerox Lisp enough to remember whether the Common Lisp side did keep comments in a similar way to InterLisp-D.

mikelevins 9y ago
There's no realistic danger of a Common Lisp implementation abandoning support for the text serialization of Lisp source code, because the reader is part of the language spec. That being the case, you can always count on support for Lisp text files, and therefore for comments stored in them. On the other hand, nothing in the standard forbids an implementation from keeping comments around in the image. Off the top of my head I don't know of an implementation that does it, but it might be a handy feature to have.
- lispm 9y ago
  Comments in Interlisp weren't really magic. Basically the comment function * was taking unevaluated args as a list. An NLAMBDA. It just returned its arg. (* this is a comment in Interlisp) This means also comments could also only be used where they fit into the Lisp syntax and not changed the control flow. In Common Lisp: (defun comment (words) words) (defmacro § (&rest words) `(comment '(,@words))) (let ((a 20)) (cond ((< 0 a 10) (§ Only when A is between 0 and 10. This is a typical value.) (* a 0.1)) (t (§ a is not between 0 and 10) (* a 0.2)))) Now it would need a special pretty printer command, so that * comment lines would be layouted such that they are aligned fit on a line. * the position of the comment should stand out from the code. Interlisp had a pretty printer macro for it. It might be better to turn that into a reader macro, so that inside the comment one would then not be constrained by s-expression syntax.... Interlisp has also the feature to store/retrieve the comment text on demand from a file. Thus the comment text might not be needed to be present in the image itself.
  - nibbula 9y ago
    I'm quite thankful that we don't have to worry about the syntactic position of comments, and the wise folks of CL, as you may know, made it so that saving comment text can be as easy as: (defparameter *comments* nil) (defun comment-saver (stream char) (declare (ignore char)) (push (list *load-pathname* (stream-file-position stream) (with-output-to-string (str) (loop :with c :while (char/= #\newline (setf c (read-char stream))) :do (write-char c str)))) *comments*) (values)) (set-macro-character #\; #'comment-saver) With apologies for boring the non-Lisp infatuated.
    - mikelevins 9y ago
      I think that after 30 years of lively interaction, my relationship with Lisp can be considered to have gone beyond infatuation. Whether that's a good thing is debatable, of course.

skybrian 9y ago

For JavaScript, the Esprima parser [1] converts JavaScript to what appears to be a JSON format. It's formalized by the Estree spec [2]. You can try it out here [3]. JSON is just regular JavaScript data structures (maps and lists). If you want type safety, there are Typescript definitions, but you could ignore that. Another example: Go has a parser and AST that comes with the standard library, which again is just using regular Go data structures (structs and interfaces). If you wanted to write a macro preprocessor, using the built-in parser seems like a much better idea than string manipulation. So, I'm wondering if there's anything more to the way Common Lisp does it? [1] http://esprima.org/ http://esprima.org/ [2] https://github.com/estree/estree https://github.com/estree/estree [3] http://esprima.org/demo/parse.html http://esprima.org/demo/parse.html

lispm 9y ago
Lisp does not use a tokenizer or a parser. It uses a reader, which reads a textual representation of data into data objects. A reader is similar to a tokenizer, but the reader just creates an internal representation of some data from an external representation. Basically what a primitive JSON reader would do. A parser would know the programming language syntax and create an internal representation of the program. The Lisp reader does not know anything about the programming language syntax. It just knows about numbers, strings, lists, arrays. But the reader does not know about conditionals, variable declarations, iteration statements, assignment statements, ... var answer = 6 * 7; (defparameter answer (* 6 7)) If we quote it, it is just Lisp data: CL-USER 8 > '(defparameter answer (* 6 7)) (DEFPARAMETER ANSWER (* 6 7)) As you can see the evaluator just returns the data, though the symbols are upcased by default. The data is not enriched as a tokenizer or parser might do. Ideally the data structure prints as it is read. There are no type or syntax annotations necessary/used. Thus it is neither a tokenizer output nor a program parse tree. In your first link the tokenizer breaks up the input in strings and categorizes it. The 42 is read into a string '42' and it is annotated as being a numeric. The Lisp reader does not do that. It reads the two characters 42 and returns an integer number object with the value 42. We can describe this data, it is a list with three elements: CL-USER 9 > (describe *) ; describe the last value (DEFPARAMETER ANSWER (* 6 7)) is a LIST 0 DEFPARAMETER ; a symbol 1 ANSWER ; another symbol 2 (* 6 7) ; a list On this level of s-expressions we know nothing about Lisp as a programming language. It is only known that DEFPARAMETER is a symbol, but not known that it is a language construct and which. It's just a symbol in the first position of a list. Since this list is also a valid Lisp program (because we wrote it as such), we can compute its value and some side-effects. This process is called evaluation. CL-USER 10 > (eval **) ; eval the second last value ANSWER ; it just returns the name of the defined variable We can also input it directly, since the READ-EVAL-PRINT-LOOP already evaluates: CL-USER 11 > (defparameter answer (* 6 7)) ANSWER As you can see, the input to the evaluator is not a tokenizer output and not a parse tree. It's just an expression tree, where the leaves are data objects. It's not (token 3 :type number) or (literal-data :type number :value 3) or something similar. It is just the 3 as a data object. The evaluator itself could be an interpreter or a compiler. The interpreter runs walks the expression tree and computes the values from the expressions, which it sees in data format. A compiler-based evaluator would take the whole expression tree, compile it and then run it.