> 3. Lack of training data > [...] > There's not an incredibly rich structure to extract, and with short outputs the weirdness doesn't compound and cycles are

> 3. Lack of training data > [...] > There's not an incredibly rich structure to extract, and with short outputs the weirdness doesn't compound and cycles aren't as likely. A common small dataset for playing with RNNs is all of Shakespeare which is somewhere in the region of 1M words. He does state that the network is trained with 2M headlines, meaning ~5-20M words. That should be enough. I would have thought that RNN would somehow work better. It would be interesting to see direct comparison of fake hacker news headlines generated with Markov chains versus RNN.