3 ms·
Recurrent networks don't give you a matrix back. They return a state machine. Qualitatively different. Transitioning from a function that returns a scalar, to
by xapata 6y ago
Recurrent networks don't give you a matrix back. They return a state machine. Qualitatively different.
Transitioning from a function that returns a scalar, to returning a polynomial, to returning an arbitrary function, and now to returning something stateful is a big difference. I suppose the next step is to write an algorithm that trains a Turing machine.
- _nhynes 6y agoRNNs provided by common frameworks really do give a matrix back (examples [0, 1]). They're about as stateful as any other object or generator function. [0]: https://pytorch.org/docs/stable/generated/torch.nn.RNN.html https://pytorch.org/docs/stable/generated/torch.nn.RNN.html [1]: https://docs.nvidia.com/deeplearning/cudnn/api/index.html#cudnnRNNForwardTraining https://docs.nvidia.com/deeplearning/cudnn/api/index.html#cu...
- xapata 6y agoWhat other statistical machine learning algorithm gives you a state machine as an output? The particular encoding -- matrix or otherwise -- is just an implementation detail.
- _nhynes 6y ago> What other statistical machine learning algorithm gives you a state machine as an output? * hidden Markov models * autoregressive models * learned LQR Sure, some of those have finite memory, but RNNs are practically limited–not as easily quantified. I see what you're saying about the single-step operation appearing unique, but I also think that RNNs can be viewed through a lens such that they look like "normal" programming concepts like generators and folding iterators.
- xapata 6y agoI'm not familiar with LQRs. Versus HMMs and linear autoregressive models, I think avoiding the need to predefine the form of the autoregression is a bigger leap than it might appear right now. I'd always been a neural network skeptic, thinking they're excessively complex for no benefit. It seems to me that's no longer true. I'm not certain that'll hold, but it's exciting.
- freeone3000 6y agoThe feedback machine is also represented as a matrix, with either implicit or explicit recurrence by the code around the model.
- xapata 6y agoYes, a finite state machine can be encoded as a matrix. Ultimately it's just a string of bits, represented by whatever physical mechanism it happens to reside in. It's still a state machine, and that's a big difference.