3 ms·

The point is that an RNN or other language model describes a stochastic process which can be seen as a Markov process with state space given by the internal sta

The point is that an RNN or other language model describes a stochastic process which can be seen as a Markov process with state space given by the internal state of the RNN. This doesn't say anything about the ease of implementing or learning such a model. Just that from a purely mathematical perspective, they are equivalent in power.

raverbashing 7y ago
Ah I see, I agree with their equivalence in the mathematical sense.
- srean 7y ago
  Am curious about what other sense of 'equivalence' you had in mind when you stated with great authority that RNNs are not Markovian.
  - raverbashing 7y ago
    In the practical sense, implementing a RNN is easier than a Markov Chain (libraries, etc) Also RNNs can "evolve" (or be adapted) more easily to other architectures like LSTMs.
    - srean 7y ago
      That line of argument does not hold as RNNs are are just special cases of hidden Markov models. If RNNs are easy to implement it means some hidden Markov models are as well. In any case by ease of implementation I think you men availability of libraries rather than something fundamental to the approach. Anyway we have strayed from your claim that RNNs are not Markovian. I was struck by the confidence in the claim especially so because it isnt true by a long shot.
    - mratsim 7y ago
      If you try implementing a RNN from scratch with the same base as the OP N-order Markov Chain and without Numpy or a deep-learning frame work, I guarantee you that you will not find it so easy. For reference his code his 400 lines and this is what he is importing: #include <stdio.h> #include <vector> #include <string> #include <unordered_map> #include <random> #include <algorithm> #include <array> We can assume that <math.h> import is reasonable for comparison To implement RNNs for text you need to implement tensors/ndarrays of 3 dimensions with slicint and all that jazz, efficient matrix multiplication, implement the RNN, implement the RNN backprop properly. RNNs stands on the shoulder of giants.

raverbashing 7y ago

Ah I see, I agree with their equivalence in the mathematical sense.

srean 7y ago
Am curious about what other sense of 'equivalence' you had in mind when you stated with great authority that RNNs are not Markovian.
- raverbashing 7y ago
  In the practical sense, implementing a RNN is easier than a Markov Chain (libraries, etc) Also RNNs can "evolve" (or be adapted) more easily to other architectures like LSTMs.
  - srean 7y ago
    That line of argument does not hold as RNNs are are just special cases of hidden Markov models. If RNNs are easy to implement it means some hidden Markov models are as well. In any case by ease of implementation I think you men availability of libraries rather than something fundamental to the approach. Anyway we have strayed from your claim that RNNs are not Markovian. I was struck by the confidence in the claim especially so because it isnt true by a long shot.
  - mratsim 7y ago
    If you try implementing a RNN from scratch with the same base as the OP N-order Markov Chain and without Numpy or a deep-learning frame work, I guarantee you that you will not find it so easy. For reference his code his 400 lines and this is what he is importing: #include <stdio.h> #include <vector> #include <string> #include <unordered_map> #include <random> #include <algorithm> #include <array> We can assume that <math.h> import is reasonable for comparison To implement RNNs for text you need to implement tensors/ndarrays of 3 dimensions with slicint and all that jazz, efficient matrix multiplication, implement the RNN, implement the RNN backprop properly. RNNs stands on the shoulder of giants.