It really depends on what units of data your chain (in this case, an n-gram model) is dealing with. You can increase or decrease the granularity and modeling d

It really depends on what units of data your chain (in this case, an n-gram model) is dealing with. You can increase or decrease the granularity and modeling depth to get different kind of erroneous data using the same training data. For example, given the same corpus, if you have an n-gram model of english characters, obviously some[1] of the words will be erroneous. But if that model is over words, none of the words will be erroneous, but the grammar will still be broken in some cases. Extend that to an n-gram over collections of grammatical phrases where the words, grammar might be correct, but it may have a nonsensical meaning in some cases, etc. [1] Obviously "some" is dependent on the amount and diversity of the training data, model detail, etc.