Curious if have you tried more traditional generative models on these data sets. It seems a even a higher order Markov or a higher order hidden Markov would per

Curious if have you tried more traditional generative models on these data sets. It seems a even a higher order Markov or a higher order hidden Markov would perform comparably. Stochastic grammars that allow recursion would be even better. With a small corpus these would run into similar difficulties but would likely perform much better than RNN/CNNs trained on the same corpus. Regardless of whether you use NN or something else your approch is the right one: start with model trained on a larger but similar corpus and then use the smaller one to modify the transition probabilities. Leaving out the "Deep" stuff will shed popular clicks on your post, probably a lot, but would get the job done with less cumulative effort. Important if for some reason you cannot use third party models. The stochastic grammars I mentioned are a lot less fiddly to train than * to vec