2 ms·

The token -8 is a bit special as well.

• eliben 13y ago
  Not sure what you mean by this. In the C standard, numeric tokens include signs, floating point dots, type specifiers (U, L, etc.), and much more. For the gory details see: https://github.com/eliben/pycparser/blob/master/pycparser/c_lexer.py https://github.com/eliben/pycparser/blob/master/pycparser/c_... But in the end, it's all in the same token.
  • marshray 13y ago
    I mean tokenizing things like "8-8" vs "8+-8" vs "8---8" present some special challenges for the lexer.