One thing I've found is that using a bit of a subset of regular expressions seems to work very nicely for doing many lexing tasks. Things like using ^[a-zA-Z][

One thing I've found is that using a bit of a subset of regular expressions seems to work very nicely for doing many lexing tasks. Things like using ^[a-zA-Z][a-zA-Z0-9]+ (or whatever shorthand your RE system has for those classes) can make a lexer nicer to work with. Now I wouldn't be trying to do lots with most of the other features in them for a lexer but it ends up a compact but simple to understand way of describing the input the lexer is looking for to pull tokens out.