4 ms·

I'm not sure how "unreasonable" the effectiveness of RNNs are if the corpus output at 2000 iterations isn't significantly better than a simple prefix based mark

I'm not sure how "unreasonable" the effectiveness of RNNs are if the corpus output at 2000 iterations isn't significantly better than a simple prefix based markov chain implementation [1] (and for the regular languages, with some extra bracket-checking), but I found the evolution visualizations really interesting. [1] http://thinkzone.wlonk.com/Gibber/GibGen.htm http://thinkzone.wlonk.com/Gibber/GibGen.htm

darkmighty 11y ago
It's quite unreasonable. He could have optimized it more for fooling humans in Gibberish generation, but that would not show the general effectiveness of the approach. The power shows (quantifiably) in compression: 1.57 bits per character wikipedia is quite hard to beat. Of course, Markov Chains are essentially universal models, so the training algorithm is the crucial distinction. I believe Markov Chains as a model quickly become inefficient (specially memory-wise) as you increase the complexity (long range correlations) of your prediction. It's an unnecessarily restrictive model for high complexity behavior that state of the art RNNs skip entirely.
- new299 11y ago
  The state of the art in compressing wikipedia is 1.278bits (on a certain subset) [1]. So that does seem pretty good. [1] http://prize.hutter1.net/ http://prize.hutter1.net/
  - jsprogrammer 11y ago
    Except this NN isn't really a compression of Wikipedia since it can only generate Wikipedia-like nonsense.
    - gwern 11y ago
      It's a compression of Wikipedia in the sense that the NN generates probability estimates of the next character given the previous; the gibberish is simply greedily asking the NN repeatedly what the most-likely next character is. However, plug it into an arithmetic coder and start feeding in an actual Wikipedia corpus, and hey presto! a pretty high performance Wikipedia compressor, which works well on Wikipedia text but not so well on other texts (like this one, with its lack of brackets).
    - teraflop 11y ago
      There's very little difference between a contextual predictive model like this and the guts of a compressor. If your prediction is good enough that you can always come up with two possible predictions for each character, each of which has a 50% chance of being correct, then obviously you can compress your input down to one bit per character by storing just enough information to tell you which choice to pick. More generally, you can use arithmetic coding to do the same thing with an arbitrary set of letter probabilities, which is exactly what you get as the output of a neural network. When the blog post says the model achieved a performance of "1.57 bits per character", that's just another way of saying "if we used the neural network as a compressor, this is how well it would perform."
      - jsprogrammer 11y ago
        I'd be interested in seeing this NN perform a lossless compression of Wikipedia at 1.57 bits per character.
murbard2 11y ago
It balances parentheses and keeps track of other long range dependencies, something markov chain implementations cannot do.
deleted 11y ago
[deleted]
pohl 11y ago
Welcome to the unbearable forced-ness of titles. Everyone's making a nod to Milan Kundera these days.
- jsprogrammer 11y ago
  http://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_of_Mathematics_in_the_Natural_Sciences http://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_...
- coldtea 11y ago
  First, it's not a nod to Kundera, but to a classic math related work that predates Kundera's book. Second, even if it was, really? As if we see plays on Kundera titles regularly on the web?
  - tim333 11y ago
    I doubt it's a reference to Kundera. I was thinking that both Eugene Wigner's 1960 article 'The Unreasonable Effectiveness of Mathematics in the Natural Sciences'[0] and Karpathy's 'The Unreasonable Effectiveness of Recurrent Neural Networks' probably touch deep aspects of the nature of existence. The first on why the universe exists and is mathematical - because at the fundamental level it is mathematical[1], and in Karpathy's case the RNNs are probably effective because they are close to the mechanisms of human consciousness. [0] Wigner's article: http://www.dartmouth.edu/~matc/MathDrama/reading/Wigner.html http://www.dartmouth.edu/~matc/MathDrama/reading/Wigner.html [1] 'physical world is completely mathematical' theory: http://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_of_Mathematics_in_the_Natural_Sciences#Max_Tegmark http://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_...
samizdatum 11y ago
That Markov Chain model operates on 4-grams by default. The RNN featured in the article generates output character-by-character, which is significantly more impressive. Here's a sample from the Markov Chain model operating on 4-grams: Ther deat is more; for in thers that undiscorns the unwortune, the pangs against a life, the law's we know no trave, the hear, thers thus pause. The only reason why it seems like the model can occasionally spell, and create anglo-sounding neologisms, is because it operates on 4-grams. Here's some character-by-character output from the same Markov Chain model. T,omotsuo ait pw,, l f,s teo efoat t hoy tha fm nwo bs rs a h enwcbr lwntikh wqmaohaaer ah es aer mkazeoltl.etnhhifcmfeifnmeeoddssmusoat irca do'ltyuntos sih i etsoatbrbdl
- mellavora 11y ago
  "do'lty untos sih i" maybe the computer was drunk?
  - stefs 11y ago
    it's a completely legit invocation for awakening cthulhu.

darkmighty 11y ago

It's quite unreasonable. He could have optimized it more for fooling humans in Gibberish generation, but that would not show the general effectiveness of the approach. The power shows (quantifiably) in compression: 1.57 bits per character wikipedia is quite hard to beat. Of course, Markov Chains are essentially universal models, so the training algorithm is the crucial distinction. I believe Markov Chains as a model quickly become inefficient (specially memory-wise) as you increase the complexity (long range correlations) of your prediction. It's an unnecessarily restrictive model for high complexity behavior that state of the art RNNs skip entirely.

new299 11y ago
The state of the art in compressing wikipedia is 1.278bits (on a certain subset) [1]. So that does seem pretty good. [1] http://prize.hutter1.net/ http://prize.hutter1.net/
- jsprogrammer 11y ago
  Except this NN isn't really a compression of Wikipedia since it can only generate Wikipedia-like nonsense.
  - gwern 11y ago
    It's a compression of Wikipedia in the sense that the NN generates probability estimates of the next character given the previous; the gibberish is simply greedily asking the NN repeatedly what the most-likely next character is. However, plug it into an arithmetic coder and start feeding in an actual Wikipedia corpus, and hey presto! a pretty high performance Wikipedia compressor, which works well on Wikipedia text but not so well on other texts (like this one, with its lack of brackets).
  - teraflop 11y ago
    There's very little difference between a contextual predictive model like this and the guts of a compressor. If your prediction is good enough that you can always come up with two possible predictions for each character, each of which has a 50% chance of being correct, then obviously you can compress your input down to one bit per character by storing just enough information to tell you which choice to pick. More generally, you can use arithmetic coding to do the same thing with an arbitrary set of letter probabilities, which is exactly what you get as the output of a neural network. When the blog post says the model achieved a performance of "1.57 bits per character", that's just another way of saying "if we used the neural network as a compressor, this is how well it would perform."
    - jsprogrammer 11y ago
      I'd be interested in seeing this NN perform a lossless compression of Wikipedia at 1.57 bits per character.

murbard2 11y ago

It balances parentheses and keeps track of other long range dependencies, something markov chain implementations cannot do.

deleted 11y ago

[deleted]

pohl 11y ago

Welcome to the unbearable forced-ness of titles. Everyone's making a nod to Milan Kundera these days.

jsprogrammer 11y ago
http://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_of_Mathematics_in_the_Natural_Sciences http://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_...
coldtea 11y ago
First, it's not a nod to Kundera, but to a classic math related work that predates Kundera's book. Second, even if it was, really? As if we see plays on Kundera titles regularly on the web?
- tim333 11y ago
  I doubt it's a reference to Kundera. I was thinking that both Eugene Wigner's 1960 article 'The Unreasonable Effectiveness of Mathematics in the Natural Sciences'[0] and Karpathy's 'The Unreasonable Effectiveness of Recurrent Neural Networks' probably touch deep aspects of the nature of existence. The first on why the universe exists and is mathematical - because at the fundamental level it is mathematical[1], and in Karpathy's case the RNNs are probably effective because they are close to the mechanisms of human consciousness. [0] Wigner's article: http://www.dartmouth.edu/~matc/MathDrama/reading/Wigner.html http://www.dartmouth.edu/~matc/MathDrama/reading/Wigner.html [1] 'physical world is completely mathematical' theory: http://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_of_Mathematics_in_the_Natural_Sciences#Max_Tegmark http://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_...

samizdatum 11y ago

That Markov Chain model operates on 4-grams by default. The RNN featured in the article generates output character-by-character, which is significantly more impressive. Here's a sample from the Markov Chain model operating on 4-grams: Ther deat is more; for in thers that undiscorns the unwortune, the pangs against a life, the law's we know no trave, the hear, thers thus pause. The only reason why it seems like the model can occasionally spell, and create anglo-sounding neologisms, is because it operates on 4-grams. Here's some character-by-character output from the same Markov Chain model. T,omotsuo ait pw,, l f,s teo efoat t hoy tha fm nwo bs rs a h enwcbr lwntikh wqmaohaaer ah es aer mkazeoltl.etnhhifcmfeifnmeeoddssmusoat irca do'ltyuntos sih i etsoatbrbdl

mellavora 11y ago
"do'lty untos sih i" maybe the computer was drunk?
- stefs 11y ago
  it's a completely legit invocation for awakening cthulhu.