4 ms·
Here is the complete scanner for a C-ish language I wrote a compiler for a while back: http://www.daimi.au.dk/~sandmann/scanner.c http://www.daimi.au.dk/~sandm
by ssp 17y ago
Here is the complete scanner for a C-ish language I wrote a compiler for a while back:
http://www.daimi.au.dk/~sandmann/scanner.c http://www.daimi.au.dk/~sandmann/scanner.c
It's 387 lines of C, and if you wanted to prove a point, you could get the line-count down by compacting the table and removing braces etc.
Note that it doesn't make any C specific assumptions. Each type of token is matched by its own function, and then in the end the longest match is chosen. This is exactly what lex does - with two differences: (a) my scanner is slower, and (b) you have the full recognition power of C instead of just regular expressions. These days (b) outweighs (a) so much that it's just not funny.
I'd agree that flex/lex are just not worth it anymore.
- nostrademons 17y agoAnd here's the lexer for a Python-ish language I'm working on: http://github.com/nostrademons/eve-language/blob/master/bootstrap_compiler/Lexer.x http://github.com/nostrademons/eve-language/blob/master/boot... That's 52 lines of rules. 68 lines of boilerplate code that was copied verbatim out of the Alex manual and changed a bit to work in a different monad. About 70 lines of code to add the indent/dedent/newline tokens for whitespace sensitivity. Note that that's with whitespace rules that are more permissive than just about any language out there (you can break lines without a continuation character between any pair of matched delimiters, as well as before or after any infix operator). Yes, hand-writing a lexer is easy. Writing one with a lexer generator is even easier (and would be easier still if Alex had an easier API to work with).