In a recent project I examined the alternatives and ended up using re2c (http://re2c.org/ http://re2c.org/). re2c's advantage over flex is that it is easy to e

In a recent project I examined the alternatives and ended up using re2c (http://re2c.org/ http://re2c.org/). re2c's advantage over flex is that it is easy to embed within existing code, as opposed to a generated yylex function that your code must interface with. re2c's advantage over a hand-written lexer is that it can be a more succinct description of the fundamental rules you're implementing and the generated code (table-driven, optionally using computed gotos) is faster. The result for switching my project was maybe 5% faster (and that was in switching from code that was hand-written to even use a lookup table in a particularly tight loop), but most importantly the result is much easier to read and modify.