3 ms·

The problem with LALR parser generators is that you end up with an LALR parser. Which is fine as long as the input is error-free, but trying to get a reasonabl

The problem with LALR parser generators is that you end up with an LALR parser. Which is fine as long as the input is error-free, but trying to get a reasonable error message out of any LALR parser is about as much fun as repeatedly poking yourself in the eye with a sharp stick. FLEX, on the other hand, is perfect for the job. I can't imagine why anyone would hand-write a lexer.

jcranmer 7y ago
> FLEX, on the other hand, is perfect for the job. I can't imagine why anyone would hand-write a lexer. Flex introduces a new dependency to your project, which involves generating source files at build time. This is a lot of complexity in your build system if this is the first time you need it. And it also by default uses a style that isn't really modern--it's more annoying to use if you want to use it in a multithreaded programming, or if you want to have multiple lexers in one program. (Or you're not even using C/C++ in the first place!). Furthermore, lexers are actually really easy to write. Most tokens [1] are recognizable within a character or two, and usually amount to a regex on the order of [a-z][a-z0-9_]+ or "([^\"]|\.)*" -- the sorts of regexes that have very trivial implementations to handcode. The lexer I wrote for a JSON-ish file format is 128 lines of code. Sure, it'd be more like 30 lines of code if I used an automatic generator, but running into any issues integrating the automatic generator into my workflow would toss out the benefits of saving 100 lines of code. [1] Excluding keywords, but keywords are usually implemented by parsing it as an identifier and looking for the identifier in a hashtable of known keywords.
beagle3 7y ago
In my experience, re2c is an improvement on lex/flex in most ways, but I agree - flex is indeed nicer than hand-rolled in every way (except if you are Arthur Whitney - his hand rolled lexers are a marvel)

jcranmer 7y ago

> FLEX, on the other hand, is perfect for the job. I can't imagine why anyone would hand-write a lexer. Flex introduces a new dependency to your project, which involves generating source files at build time. This is a lot of complexity in your build system if this is the first time you need it. And it also by default uses a style that isn't really modern--it's more annoying to use if you want to use it in a multithreaded programming, or if you want to have multiple lexers in one program. (Or you're not even using C/C++ in the first place!). Furthermore, lexers are actually really easy to write. Most tokens [1] are recognizable within a character or two, and usually amount to a regex on the order of [a-z][a-z0-9_]+ or "([^\"]|\.)*" -- the sorts of regexes that have very trivial implementations to handcode. The lexer I wrote for a JSON-ish file format is 128 lines of code. Sure, it'd be more like 30 lines of code if I used an automatic generator, but running into any issues integrating the automatic generator into my workflow would toss out the benefits of saving 100 lines of code. [1] Excluding keywords, but keywords are usually implemented by parsing it as an identifier and looking for the identifier in a hashtable of known keywords.

beagle3 7y ago

In my experience, re2c is an improvement on lex/flex in most ways, but I agree - flex is indeed nicer than hand-rolled in every way (except if you are Arthur Whitney - his hand rolled lexers are a marvel)