4 ms·
What is a feed forward network? Is that anything like the current-state and next-state functions for representing finite state machines?
by calf 4y ago
What is a feed forward network? Is that anything like the current-state and next-state functions for representing finite state machines?
- telotortium 4y agoNo, a feed forward network is just a neural network which doesn't contain cycles, in contrast to recurrent neural networks, which do. A system that can't perform cyclical behavior is not Turing complete, since we know that it will halt. Large language models are not recurrent networks because they don't contain cycles (the whole point of the transformer architecture is to mix position information with the tokens so that the model is sensitive to short and long range context, which obviates the need for recurrent networks for "memory" ), so they are not Turing complete - at least in a single iteration - and will probably never compute many functions efficiently. Nevertheless, because you can feed the generated text back into the LLM pipeline, they can operate on current-state and next-state as in your example of a finite state machine.
- calf 4y agoYeah that is a neat hack one of the comments mentioned. Also if a model of computation that is not Turing complete can apparently do English, then what does that say about the computational complexity of English, this piques my curiosity.