> Secondly, you don't have to have a separate scanner. That's sort of the point—supporting "eggular expressions" is easy because tokenizing them is drastically

> Secondly, you don't have to have a separate scanner. That's sort of the point—supporting "eggular expressions" is easy because tokenizing them is drastically simpler than the hacks regarding context-sensitivity that you must weave into both your parser and scanner (and complicate the scanner–parser contract) if you want conventional unparenthesized literals. Hm, I'm not sure I understand this bit. It seems bad to use the same scanner for the tokens inside your regular expression as for the rest of your language. Two examples: 1. In my langauge, "3.14" is a token, but when parsing an eggex you'd want that to be at least 3 tokens because the "." is typically your "match one of any character" token. 2. My language doesn't have a "?" operator, but if we're using the same scanner for eggexes as the rest of the language, I'd have to include a token for that to support "zero or one instances" in the eggexes. Sure, your main scanner can (and probably should) just grab everything between "(/" and "/)" as one token, but there's no avoiding going over the contents of the eggex with a scanner. I'm not saying you have to scan it during the regular scanning--that's probably not a good idea--but you're going to have scan it at some point in order to compile the eggex.