2 ms·

That's from the programmer point of view an implementation detail. The main interface for the programmer is the function READ and it returns data, not a stream

• kazinator 9y ago
  Although there is no program-visible "token" data type, the specification describes tokens and uses that word. Since the reader is available to programs, programs can be written to test hypotheses about the implementation's treatment of tokens; tokens can be composed as character strings, passed into read-from-string and the results of the operation can reveal something about the tokenizing black box, without yielding the tokens themselves. (That said, the Common Lisp syntax isn't completely treated as a token stream by the reader, in the glossary sense of "token", since that word is defined only as the read syntax of numbers and symbols; thus string literals and other bit of notation aren't defined as tokens.)
  • lispm 9y ago
    Exactly, it reads single tokens in some places and then creates data from them. It uses the word token for that, but that's arbitrary. That's not what a Tokenizer does, which usually would create a stream of tokens for all the elements in the input. A token would be a string with some metadata.