The split in lexing and parsing is indeed artificial and one often has to make use of states or even cases where the parsing is controlling the lexing. Try for

The split in lexing and parsing is indeed artificial and one often has to make use of states or even cases where the parsing is controlling the lexing. Try for example to parse an HTML file with embedded CSS and/or JavaScrips using lex(flex) and yacc(bison). Recursive decent, where the parser asks the 'scanner' to accept a certain symbol at a certain location in the input stream, resolves most of these problems, such as for example the fact that in HTML the tags are case insensitive while JavaScript the keywords are case sensitive. If you try to develop a parser that is user-friendly, but maybe not the most efficient (if you allow back-tracking) the whole problem of parsing become far less complicated. (There a good methods to counter performance problems due to back-tracking.)