Interesting article... I wrote a lexer for python as part of gpython: https://github.com/go-python/gpython/blob/f100534592c96b7922c59660553ee77fbf217da1/parser

Interesting article... I wrote a lexer for python as part of gpython: https://github.com/go-python/gpython/blob/f100534592c96b7922c59660553ee77fbf217da1/parser/lexer.go#L375 https://github.com/go-python/gpython/blob/f100534592c96b7922... It is remarkably complex and has a huge amount of state including separate indent levels for parentheses, brackets and braces! The lexer is extremely well defined in the python docs; I wrote my version entirely by looking at the docs and not at the CPython source code (unlike let's say the compiler!). Shoving the complexity into the lexer does make the python grammar itself quite straight forward though.