2 ms·
Somewhat related, I recently read about the undocumented tokenizer that comes with the python re module. https://www.reddit.com/r/ProgrammerTIL/comments/4tpt03
by tyingq 5y ago
Somewhat related, I recently read about the undocumented tokenizer that comes with the python re module. https://www.reddit.com/r/ProgrammerTIL/comments/4tpt03/python_til_about_rescanner_which_is_useful_for/ https://www.reddit.com/r/ProgrammerTIL/comments/4tpt03/pytho...
- yatac42 5y agoWord of warning: The Scanner class does not adhere to the maximal munch rule, so you shouldn't use it to match keywords the way that it's done in the code in the linked reddit thread. If you replace the word "foobar" in the input with "truebar", it will then be tokenized as the keyword true followed by the identifier "bar", which is obviously not what you want.