You can punt as much into the parsing stage as you like, of course. It's not like there's even any requirement that the two stages be separated. All of the stu

You can punt as much into the parsing stage as you like, of course. It's not like there's even any requirement that the two stages be separated. All of the stuff I mentioned gives you the choice of either more work during parsing (which is usually hard enough already), or building a crazy Rube Goldberg-esque tokenizer--neither of which is appealing.