If you feed it words and appropriate features from a programming language you should get something close to a tokenizer. Laura dietz ( http://www.mpi-inf.mpg.de

If you feed it words and appropriate features from a programming language you should get something close to a tokenizer. Laura dietz ( http://www.mpi-inf.mpg.de/~dietz/index.html#publications http://www.mpi-inf.mpg.de/~dietz/index.html#publications ) has some work that applies similar techniques to programming languages to find bugs.