3 ms·
Lisp does not use a tokenizer or a parser. It uses a reader, which reads a textual representation of data into data objects. A reader is similar to a tokenizer
by lispm 9y ago
Lisp does not use a tokenizer or a parser. It uses a reader, which reads a textual representation of data into data objects.
A reader is similar to a tokenizer, but the reader just creates an internal representation of some data from an external representation. Basically what a primitive JSON reader would do.
A parser would know the programming language syntax and create an internal representation of the program. The Lisp reader does not know anything about the programming language syntax. It just knows about numbers, strings, lists, arrays. But the reader does not know about conditionals, variable declarations, iteration statements, assignment statements, ...
var answer = 6 * 7;
(defparameter answer (* 6 7))
If we quote it, it is just Lisp data:
CL-USER 8 > '(defparameter answer (* 6 7))
(DEFPARAMETER ANSWER (* 6 7))
As you can see the evaluator just returns the data, though the symbols are upcased by default. The data is not enriched as a tokenizer or parser might do. Ideally the data structure prints as it is read. There are no type or syntax annotations necessary/used. Thus it is neither a tokenizer output nor a program parse tree. In your first link the tokenizer breaks up the input in strings and categorizes it. The 42 is read into a string '42' and it is annotated as being a numeric. The Lisp reader does not do that. It reads the two characters 42 and returns an integer number object with the value 42.
We can describe this data, it is a list with three elements:
CL-USER 9 > (describe *) ; describe the last value
(DEFPARAMETER ANSWER (* 6 7)) is a LIST
0 DEFPARAMETER ; a symbol
1 ANSWER ; another symbol
2 (* 6 7) ; a list
On this level of s-expressions we know nothing about Lisp as a programming language. It is only known that DEFPARAMETER is a symbol, but not known that it is a language construct and which. It's just a symbol in the first position of a list.
Since this list is also a valid Lisp program (because we wrote it as such), we can compute its value and some side-effects. This process is called evaluation.
CL-USER 10 > (eval **) ; eval the second last value
ANSWER ; it just returns the name of the defined variable
We can also input it directly, since the READ-EVAL-PRINT-LOOP already evaluates:
CL-USER 11 > (defparameter answer (* 6 7))
ANSWER
As you can see, the input to the evaluator is not a tokenizer output and not a parse tree. It's just an expression tree, where the leaves are data objects. It's not (token 3 :type number) or (literal-data :type number :value 3) or something similar. It is just the 3 as a data object.
The evaluator itself could be an interpreter or a compiler. The interpreter runs walks the expression tree and computes the values from the expressions, which it sees in data format. A compiler-based evaluator would take the whole expression tree, compile it and then run it.
- kazinator 9y agoThere is definitely a tokenizer in Common Lisp, and one that is pretty much "sealed off" to the programmer. The current read table can assign a character to be a "constituent". Constituent characters are gathered by the reader into a token. Then they are analyzed. A sequence of unescaped digits like 12345 becomes an integer. Floating-point literals are recognized. A symbol token is analyzed for the package : or :: . And so on.
- lispm 9y agoThat's from the programmer point of view an implementation detail. The main interface for the programmer is the function READ and it returns data, not a stream of tokens.
- kazinator 9y agoAlthough there is no program-visible "token" data type, the specification describes tokens and uses that word. Since the reader is available to programs, programs can be written to test hypotheses about the implementation's treatment of tokens; tokens can be composed as character strings, passed into read-from-string and the results of the operation can reveal something about the tokenizing black box, without yielding the tokens themselves. (That said, the Common Lisp syntax isn't completely treated as a token stream by the reader, in the glossary sense of "token", since that word is defined only as the read syntax of numbers and symbols; thus string literals and other bit of notation aren't defined as tokens.)
- lispm 9y agoExactly, it reads single tokens in some places and then creates data from them. It uses the word token for that, but that's arbitrary. That's not what a Tokenizer does, which usually would create a stream of tokens for all the elements in the input. A token would be a string with some metadata.