4 ms·
This is normal for lexers in whitespace-insensitive languages. In spirit this is similar to allowing "foo+" (plus after "foo") or any other token concatenation.
by eliben 13y ago
This is normal for lexers in whitespace-insensitive languages. In spirit this is similar to allowing "foo+" (plus after "foo") or any other token concatenation.
If this is an error in the language, the parser easily catches it.
Some languages AFAIR allow "units" to be attached to numbers so some use cases are actually valid. In any case, it's a higher-level decision the parser makes.
- mikeash 13y agoC, for example, allows trailing letters to specify the type of a numeric literal: 42 // int 42U // unsigned int 42LL // long long 42ULL // unsigned long long 42.0 // double 42.0f // float Edit: except that whitespace isn't allowed here. "42 LL" is not legal. So that's kind of the opposite case!
- eliben 13y agoIn the case of C it's a bit special because that suffix is usually handled by the lexer - i.e. there are no production rules for it in the C grammar. This is also why a whitespace is not allowed - it's not a single token any longer!
- drivebyacct2 13y agoWouldn't that presumably be implementation specific?
- marshray 13y agoThe token -8 is a bit special as well.
- eliben 13y agoNot sure what you mean by this. In the C standard, numeric tokens include signs, floating point dots, type specifiers (U, L, etc.), and much more. For the gory details see: https://github.com/eliben/pycparser/blob/master/pycparser/c_lexer.py https://github.com/eliben/pycparser/blob/master/pycparser/c_... But in the end, it's all in the same token.
- marshray 13y agoI mean tokenizing things like "8-8" vs "8+-8" vs "8---8" present some special challenges for the lexer.