5 ms·

Full Python 3 Grammar Specification

• xg15 8y ago
  I was wondering how they'd specify a language with semantic indents in an EBNF grammar. Apparently they don't do that at all - they seem to use "virtual" INDENT and DEDENT tokens that act like brackets. I suppose there is an instruction written in prose somewhere how to convert the actual python into the python-with-invisible-brackets that is defined here. Very clever!
  • kbirkeland 8y ago
    The algorithm for generating those tokens is explained in the Lexical Analysis reference[0], and a quick look at the cpython source shows the logic is implemented in Parser/tokenizer.c[1]. [0]: https://docs.python.org/3/reference/lexical_analysis.html#indentation https://docs.python.org/3/reference/lexical_analysis.html#in... [1]: https://github.com/python/cpython/blob/master/Parser/tokenizer.c https://github.com/python/cpython/blob/master/Parser/tokeniz...
    • xg15 8y ago
      Very cool! Thanks much for the links.