4 ms·

Isn't the author's definition of RNNs wrong? I thought the difference is that a RNN allows connection back to previous layers, compared to a feed-forward net.

• fpgaminer 11y ago
  Karpathy usually talks about machine learning topics from multiple viewpoints, and usually (in my experience with his writings) prefers more loose, non-traditional interpretations (that ultimately lead to better understanding of the underlying mechanics of the approach). In this case, his point was that one way RNNs differ from FFNNs is their ability to accept arbitrarily sized inputs and generated arbitrarily sized outputs. That's pretty important, which is likely why he emphasizes it. But the rest of the article shows the salient point; RNNs are NNs that hold a state vector. Saying that RNNs are NNs that allow connections back to previous layers is true, but that's only one way of looking at it. Holding state is another, since it implies backwards connections. Feedback is another term. And because they have backwards connections, state, feedback, etc, they also posses the capacity to handle non-fixed sized inputs and outputs. In summary; it's different viewpoints of the same mathematical object. Karpathy focuses on the ability of RNNs to handle arbitrarily long inputs and outputs, because that's something FFNNs cannot do.