3 ms·

Not the author, just wrote a Python parser recently. The problem with most grammar specifications is that they are not clear about the tokenization stage and as

Not the author, just wrote a Python parser recently. The problem with most grammar specifications is that they are not clear about the tokenization stage and assume that you already have a valid token stream to work with, which is of course not the case usually. The Python grammar specification for example does not even mention anything about how one should implement the tokenizer (to my knowledge). That said, most programming languages can be formulated in an almost context-free way. For real-world parsers, I think speed is just valued much higher than elegance, which often makes sense, especially for scripting languages where the parsing speed is directly proportional to program startup time (leaving caching aside).

chubot 10y ago
Agreed, although I would not say "most" can be formulated as CFGs. There is the lexer issue you mention, and that is mentioned in that article, but there are also tons of other issues in real languages. Languages that are close to context free: Python, Java, JavaScript, R, Go, awk. (and I think Standard ML and OCaml) Languages that are not even close: C, C++, Perl, Unix shell, make. Languages I'm not sure about but which probably fall in the "no" category: PHP, Ruby, C#. So I think that is enough to warrant some research into parsing tools for non-CFGs. (PEGs are interesting and I explored them for some of these applications)