2 ms·
It's memoryless because it's a pure function. All feedforward neural networks are memoryless. This is as opposed to an RNN, which has an internal state.
by Legend2440 2y ago
It's memoryless because it's a pure function. All feedforward neural networks are memoryless.
This is as opposed to an RNN, which has an internal state.
- ben_w 2y agoWhile this is absolutely correct when the RNN has ℝeal-valued internal state, in concrete implementations the internal state in an RNN has some number of bits and one can draw a Markov graph with a group of 2^n variants of states to represent all possible states of n bits in some part of the system. Likewise the context window of an LLM can be considered a state with some number of bits. It's just more explicit in an LLM's architecture vs. a RNN's runtime that the number of bits is finite.