That depends a lot on your parser. Personally if I write a parser I don't generally bother with having a separate tokenizing/lexing stage. Tokenizing happens as

That depends a lot on your parser. Personally if I write a parser I don't generally bother with having a separate tokenizing/lexing stage. Tokenizing happens as-needed and inline.