The lexer isn't context-free, and you should re-read the paragraph. The part that you left out in the dots is crucial: > The Python lexer keeps, in addition, a

The lexer isn't context-free, and you should re-read the paragraph. The part that you left out in the dots is crucial: > The Python lexer keeps, in addition, a stack of counters for the indentation levels. Moreover, it keeps track of the nesting of parentheses; and the language of balanced parentheses is the canonical non-regular, context-free language! So in this design, there is no clear separation of the context-free part and the context-sensitive part, and the context-sensitive part goes well beyond what a typical lexer can do.