3 ms·

Strangely enough, I find the output of Markov Chain - type text generators much more readable and coherent than the output of those RNNs. A Marov Chain text gen

Strangely enough, I find the output of Markov Chain - type text generators much more readable and coherent than the output of those RNNs. A Marov Chain text generator is also easy to implement and understand.

visakanv 10y ago
Interesting. Why do you think that is? (More readable, that is)
- aninhumer 10y ago
  If I were going to guess (not the parent), it's because human parsing is quite localised, and Markov models tend to optimise well for localised coherence. Their failure mode is long sentences which don't go anywhere (or mean anything), but that's less grating than a blatant parse failure, especially if you're skim reading.
  - visakanv 10y ago
    That actually makes a lot of sense, great point.
- marktangotango 10y ago
  Claude Shannon[1], information theory guy, worked on machine generated music quite a lot. Some of the ideas transfer. Information theory informs a lot in this area[2]. [1] https://en.wikipedia.org/wiki/Claude_Shannon https://en.wikipedia.org/wiki/Claude_Shannon [2] http://www.music-cog.ohio-state.edu/Music829D/Notes/Infotheory.html http://www.music-cog.ohio-state.edu/Music829D/Notes/Infotheo...
  - visakanv 10y ago
    Fascinating!
Infernal 10y ago
I wonder how hard it would be to use a Markov chain as a syntactic filter on the output of the RNNs?
umlaut 10y ago
The interesting properties of RNNs aren't in readability. RNNs can learn syntactical states, e.g. "this text is in quotes" that enable some really cool analysis tricks. OpenAI published an article about this recently, where they showed that an RNN, without being trained on it, learned the difference between positive and negative reviews. The fact that an RNN encodes sentiment is an incredibly useful feature. source: https://blog.openai.com/unsupervised-sentiment-neuron/ https://blog.openai.com/unsupervised-sentiment-neuron/
- sharanvkaur 10y ago
  thanks for bringing this up! i saw this article last week and it THRILLED me. so much potential here.

visakanv 10y ago

Interesting. Why do you think that is? (More readable, that is)

aninhumer 10y ago
If I were going to guess (not the parent), it's because human parsing is quite localised, and Markov models tend to optimise well for localised coherence. Their failure mode is long sentences which don't go anywhere (or mean anything), but that's less grating than a blatant parse failure, especially if you're skim reading.
- visakanv 10y ago
  That actually makes a lot of sense, great point.
marktangotango 10y ago
Claude Shannon[1], information theory guy, worked on machine generated music quite a lot. Some of the ideas transfer. Information theory informs a lot in this area[2]. [1] https://en.wikipedia.org/wiki/Claude_Shannon https://en.wikipedia.org/wiki/Claude_Shannon [2] http://www.music-cog.ohio-state.edu/Music829D/Notes/Infotheory.html http://www.music-cog.ohio-state.edu/Music829D/Notes/Infotheo...
- visakanv 10y ago
  Fascinating!

Infernal 10y ago

I wonder how hard it would be to use a Markov chain as a syntactic filter on the output of the RNNs?

umlaut 10y ago

The interesting properties of RNNs aren't in readability. RNNs can learn syntactical states, e.g. "this text is in quotes" that enable some really cool analysis tricks. OpenAI published an article about this recently, where they showed that an RNN, without being trained on it, learned the difference between positive and negative reviews. The fact that an RNN encodes sentiment is an incredibly useful feature. source: https://blog.openai.com/unsupervised-sentiment-neuron/ https://blog.openai.com/unsupervised-sentiment-neuron/

sharanvkaur 10y ago
thanks for bringing this up! i saw this article last week and it THRILLED me. so much potential here.