4 ms·
In the case of C it's a bit special because that suffix is usually handled by the lexer - i.e. there are no production rules for it in the C grammar. This is al
by eliben 13y ago
In the case of C it's a bit special because that suffix is usually handled by the lexer - i.e. there are no production rules for it in the C grammar. This is also why a whitespace is not allowed - it's not a single token any longer!
- drivebyacct2 13y agoWouldn't that presumably be implementation specific?
- marshray 13y agoThe token -8 is a bit special as well.
- eliben 13y agoNot sure what you mean by this. In the C standard, numeric tokens include signs, floating point dots, type specifiers (U, L, etc.), and much more. For the gory details see: https://github.com/eliben/pycparser/blob/master/pycparser/c_lexer.py https://github.com/eliben/pycparser/blob/master/pycparser/c_... But in the end, it's all in the same token.
- marshray 13y agoI mean tokenizing things like "8-8" vs "8+-8" vs "8---8" present some special challenges for the lexer.