3 ms·
As an addendum: In some languages (who will remain unnamed) lexical analysis is actually a pain due to funny whitespace rules or syntax-dependent tokenizing. Fo
by camccann 17y ago
As an addendum: In some languages (who will remain unnamed) lexical analysis is actually a pain due to funny whitespace rules or syntax-dependent tokenizing. For instance, something like:
* whitespace around operators is optional, except that sometimes a symbol adjacent to a variable means something completely different
* "foo" is a keyword in type signatures, but a normal identifier otherwise
* indentation is meaningful, and must generate INDENT or DEDENT tokens depending on the indentation of some arbitrary number of prior lines
* indentation is meaningful based on whether the first non-whitespace token on a line is aligned in the same column as a specific token in the prior line, based on parsed syntax, oh god what were these people smoking
...and so on. It's easy to get into situations where merely lexing requires keeping some sort of context, requires interleaving with parsing such that parse errors may cause the lexer to backtrack and reinterpret things, or possibly even worse.
In fact, I wouldn't be surprised if it were possible to construct an especially pathological bit of perl that couldn't be lexically analyzed without running the program.
- statictype 17y ago>* "foo" is a keyword in type signatures, but a normal identifier otherwise Isn't this decision something that should be punted out of the tokenizer and into the parser?
- camccann 17y agoYou can punt as much into the parsing stage as you like, of course. It's not like there's even any requirement that the two stages be separated. All of the stuff I mentioned gives you the choice of either more work during parsing (which is usually hard enough already), or building a crazy Rube Goldberg-esque tokenizer--neither of which is appealing.
- tptacek 17y agoI'd rather complicate a lexer a little bit than debug parser conflicts.
- chromatic 17y ago> I wouldn't be surprised if it were possible to construct an especially pathological bit of perl that couldn't be lexically analyzed without running the program. Yes and no. It's trivial to construct a pathological Perl 5 program which has indeterminate parsing without running... but in those cases where it's possible, you have only one of two or three possible interpretations for a specific sequence of tokens. As well, no one would write code that indeterminate by accident. It's always malicious.