What you are doing is essentially different: you are doing word sequencing and possibly part-of-speech tagging from a character stream. What is being done in th

What you are doing is essentially different: you are doing word sequencing and possibly part-of-speech tagging from a character stream. What is being done in this post, is parsing, which imposes a tree-like structure on the input sequence. These two techniques complement each other, and it might be beneficial to first do the word sequencing (the Viterbi algorithm is a good choice) and parsing after that.