FWIW there is an example of building a tokenizer/scanner using documented features of the re module here: https://docs.python.org/3.11/library/re.html#writing-a

FWIW there is an example of building a tokenizer/scanner using documented features of the re module here: https://docs.python.org/3.11/library/re.html#writing-a-tokenizer https://docs.python.org/3.11/library/re.html#writing-a-token... re.Scanner looks more succinct though...!