3 ms·
In cases like this, the parser and lexer can often produce a better error message if they are written to accept a more lax input and then check it for errors.
by ufo 7y ago
In cases like this, the parser and lexer can often produce a better error message if they are written to accept a more lax input and then check it for errors.
For example, instead of faithfully implementing the grammar from the specification, allow numbers with leading zeroes and then produce an error for them.
Another situation where this comes up is parsing language keywords. Instead of writing a separate lexer rule for every keyword, write a single rule for keyword-or-identifier, and then use a hash table lookup inside of that to determine if it is a keyword or identifier.
- repsilat 7y agoThis seems more flexible to me somehow. Using the approach in TFA, can the laser handle tokens that are prefixes of other tokens? Or even tokens that share prefixes? Lexing "truefalse" as two adjacent tokens "true" and "false" seems slightly crazier than just lexing it as one (meaningless) token.
- ufo 7y agoUsually, if there is a token that is a prefix of another the longer token wins. For example, in Javascript --x is parsed as the decrement operator instead of as two unary "-" operators.
- raverbashing 7y ago> instead of faithfully implementing the grammar from the specification, allow numbers with leading zeroes and then produce an error for them. But that's the problem. The tokenizer doesn't talk to the grammar parser (and vice versa) The tokenizer could understand numbers with leading zeroes and throw an error there. Something to think about: do languages - not json - interpret -1.2 as [MINUS][NUMBER] or just [NUMBER]. Or how does languages deal with 1.0-2.0 compared to 1.0+-2.0
- ufo 7y agoExactly. I was referring to the grammar the tokenizer uses for tokenizing numbers. Instead of faithfully copying the JSON spec, you could use a simpler rule that also accepts leading zeros. As for the leading "-", in languages that have expressions it is common to parse the "-" as a prefix operator because that covers both negative numbers (-1.0) and negating variables (-x). But in JSON there are no expressions like 1.0-2.0 so the leading "-" is parsed as part of the number.
- zwkrt 7y agoIf you know you are expecting an expression, then it is easy for the parser to understand ‘-‘ as a unary negation if it is the first symbol, otherwise it must be a binary subtraction operator. Expr: ‘-‘ Expr { return -1 * $2; } | Expr ‘-‘ Expr { return $1 - $2; } | ‘(‘ Expr ‘)’ { return $2; } | ...
- deleted 7y ago[deleted]