3 ms·
> So how would you tokenize without the use of regular expressions? Since this need doesn't appear to be an everyday one, with clearly defined targets, a simpl
by shakna 7y ago
> So how would you tokenize without the use of regular expressions?
Since this need doesn't appear to be an everyday one, with clearly defined targets, a simple hand-written lexer isn't hard to write, and will make less mistakes than a regex. Just use a scanning approach. As a bonus, you'll still be able to read in 12 months time.
- unlinkr 7y agoWhy would a hand-written lexer have fewer mistakes than a regular expression using an off-the-shelf regex engine? They would need to encode the same lexical grammar, so at that level there is the same amount of complexity. Writing a lexer by hand is just trading ten lines of regex (a widely known declarative DSL) with hundreds of lines of custom code. I don't see how that would be more maintainable in the long run.